Sep 17 15:30:12.896228 ip-10-0-128-15 systemd[1]: Starting Kubernetes Kubelet... Sep 17 15:30:13.244589 ip-10-0-128-15 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 15:30:13.244589 ip-10-0-128-15 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 15:30:13.244589 ip-10-0-128-15 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 15:30:13.244589 ip-10-0-128-15 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 15:30:13.244589 ip-10-0-128-15 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 15:30:13.246201 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.246115 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 15:30:13.253335 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253318 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 15:30:13.253335 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253335 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253339 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253343 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253347 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253350 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253353 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253356 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253359 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253362 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253367 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253369 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253373 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253382 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253385 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253388 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253391 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253394 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253397 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253400 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253402 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 15:30:13.253398 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253405 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253408 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253411 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253414 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253417 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253420 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253422 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253425 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253428 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253431 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253434 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253437 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253439 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253442 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253444 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253447 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253449 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253452 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253454 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253457 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 15:30:13.254119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253459 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253462 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253465 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253467 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253469 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253473 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253475 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253478 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253480 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253483 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253485 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253488 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253490 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253493 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253496 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253499 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253503 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253506 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253510 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 15:30:13.254895 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253514 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253517 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253520 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253523 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253526 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253528 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253531 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253533 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253536 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253539 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253542 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253544 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253547 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253549 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253552 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253555 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253557 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253559 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253562 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253565 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253568 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 15:30:13.255444 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253571 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253574 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253576 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253582 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253585 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253589 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253594 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253598 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253601 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253604 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253606 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253609 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253612 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253615 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253618 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253621 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253623 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 15:30:13.256070 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.253626 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 15:30:13.256994 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.256974 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 15:30:13.256994 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.256992 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 15:30:13.256994 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.256997 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257002 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257006 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257012 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257017 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257021 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257025 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257029 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257033 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257037 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257041 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257045 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257049 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257053 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257060 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257065 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257070 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257074 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257078 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 15:30:13.257163 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257084 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257088 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257093 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257097 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257101 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257106 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257110 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257115 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257119 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257123 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257128 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257133 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257138 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257142 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257146 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257151 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257155 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257159 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257163 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 15:30:13.258028 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257167 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257171 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257178 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257185 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257192 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257196 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257201 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257205 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257211 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257215 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257219 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257223 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257228 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257232 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257236 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257241 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257245 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257249 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257254 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257259 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 15:30:13.258856 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257264 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257268 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257272 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257276 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257281 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257285 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257291 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257295 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257300 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257304 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257309 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257313 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257318 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257322 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257327 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257331 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257335 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257339 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257343 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257349 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 15:30:13.259734 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257355 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257362 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257367 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257373 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257377 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257382 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257386 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257390 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257394 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257399 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257404 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257408 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257413 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257417 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257422 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257426 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257430 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257434 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.257439 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258109 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 15:30:13.260466 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258119 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258124 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258130 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258137 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258142 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258147 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258152 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258156 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258161 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258165 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258169 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258174 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258177 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258181 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258186 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258192 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258196 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258201 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258205 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 15:30:13.261055 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258209 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258213 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258218 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258221 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258225 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258231 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258235 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258240 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258245 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258249 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258253 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258257 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258262 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258266 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258271 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258275 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258280 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258284 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258288 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258292 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 17 15:30:13.262104 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258296 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258301 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258305 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258310 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258314 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258318 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258323 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258327 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258331 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258337 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258341 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258345 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258349 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258356 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258361 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258367 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258371 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258376 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258381 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258385 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 15:30:13.262711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258390 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258395 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258399 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258407 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258412 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258417 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258422 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258426 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258430 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258435 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258439 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258443 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258448 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258452 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258456 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258460 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258464 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258469 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258473 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 15:30:13.263401 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258477 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258481 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258487 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258492 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258497 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258501 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258505 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258510 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258514 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258519 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258523 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258527 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258531 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258536 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258540 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258544 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258553 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258557 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258562 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.258566 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 15:30:13.263864 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258686 2581 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258742 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258752 2581 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258759 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258766 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258771 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258779 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258785 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258791 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258796 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258801 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258806 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258811 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258816 2581 flags.go:64] FLAG: --cgroup-root="" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258821 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258826 2581 flags.go:64] FLAG: --client-ca-file="" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258832 2581 flags.go:64] FLAG: --cloud-provider="external" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258837 2581 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258844 2581 flags.go:64] FLAG: --cluster-domain="" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258849 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258853 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258858 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258863 2581 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258869 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 15:30:13.264497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258874 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258879 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258884 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258889 2581 flags.go:64] FLAG: --contention-profiling="false" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258897 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258902 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258907 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258928 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258935 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258939 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258944 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258949 2581 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258953 2581 flags.go:64] FLAG: --enable-server="true" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258958 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258966 2581 flags.go:64] FLAG: --event-burst="100" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258971 2581 flags.go:64] FLAG: --event-qps="50" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258975 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258980 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258985 2581 flags.go:64] FLAG: --eviction-hard="" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258991 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.258996 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259001 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259006 2581 flags.go:64] FLAG: --eviction-soft="" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259011 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259016 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 15:30:13.265156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259031 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259036 2581 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259041 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259046 2581 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259051 2581 flags.go:64] FLAG: --feature-gates="" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259057 2581 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259061 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259066 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259071 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259076 2581 flags.go:64] FLAG: --healthz-port="10248" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259081 2581 flags.go:64] FLAG: --help="false" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259088 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259094 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259098 2581 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259103 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259109 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259114 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259119 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259124 2581 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259128 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259133 2581 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259138 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259143 2581 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259148 2581 flags.go:64] FLAG: --kube-reserved="" Sep 17 15:30:13.265786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259153 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259157 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259162 2581 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259169 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259174 2581 flags.go:64] FLAG: --lock-file="" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259179 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259184 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259188 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259197 2581 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259202 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259207 2581 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259212 2581 flags.go:64] FLAG: --logging-format="text" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259217 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259222 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259227 2581 flags.go:64] FLAG: --manifest-url="" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259231 2581 flags.go:64] FLAG: --manifest-url-header="" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259238 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259243 2581 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259249 2581 flags.go:64] FLAG: --max-pods="110" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259256 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259261 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259266 2581 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259272 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259278 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259282 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 15:30:13.266405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259287 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259300 2581 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259305 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259310 2581 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259315 2581 flags.go:64] FLAG: --pod-cidr="" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259319 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259328 2581 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259333 2581 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259337 2581 flags.go:64] FLAG: --pods-per-core="0" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259342 2581 flags.go:64] FLAG: --port="10250" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259349 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259353 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fdea12022a699a1d" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259359 2581 flags.go:64] FLAG: --qos-reserved="" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259363 2581 flags.go:64] FLAG: --read-only-port="10255" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259368 2581 flags.go:64] FLAG: --register-node="true" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259373 2581 flags.go:64] FLAG: --register-with-taints="" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259379 2581 flags.go:64] FLAG: --registry-burst="10" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259385 2581 flags.go:64] FLAG: --registry-qps="5" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259390 2581 flags.go:64] FLAG: --reserved-cpus="" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259394 2581 flags.go:64] FLAG: --reserved-memory="" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259400 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259406 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259411 2581 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259416 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259421 2581 flags.go:64] FLAG: --runonce="false" Sep 17 15:30:13.267092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259425 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259431 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259437 2581 flags.go:64] FLAG: --seccomp-default="false" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259442 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259447 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259452 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259457 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259461 2581 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259467 2581 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259471 2581 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259476 2581 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259481 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259485 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259490 2581 flags.go:64] FLAG: --system-cgroups="" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259495 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259503 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259508 2581 flags.go:64] FLAG: --tls-cert-file="" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259514 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259520 2581 flags.go:64] FLAG: --tls-min-version="" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259525 2581 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259529 2581 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259534 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259539 2581 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259544 2581 flags.go:64] FLAG: --v="2" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259550 2581 flags.go:64] FLAG: --version="false" Sep 17 15:30:13.267726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259558 2581 flags.go:64] FLAG: --vmodule="" Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259564 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.259569 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259750 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259757 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259761 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259766 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259770 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259775 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259779 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259783 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259787 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259791 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259795 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259799 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259803 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259808 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259812 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259816 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259820 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 15:30:13.268414 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259824 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259829 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259834 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259838 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259844 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259849 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259853 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259857 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259861 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259865 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259869 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259874 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259878 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259883 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259888 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259893 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259899 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259905 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259929 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259934 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 15:30:13.268980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259938 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259942 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259946 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259950 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259954 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259959 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259963 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259967 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259971 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259975 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259980 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259984 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259988 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259993 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.259997 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260001 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260010 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260016 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260020 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260025 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 15:30:13.269470 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260029 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260034 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260039 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260044 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260048 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260053 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260059 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260063 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260067 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260072 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260077 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260081 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260086 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260090 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260094 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260098 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260102 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260106 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260110 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 15:30:13.269980 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260114 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260119 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260123 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260127 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260132 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260136 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260140 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260144 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260148 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260154 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260158 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260163 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260167 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260171 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260175 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260180 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260184 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260188 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260192 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260196 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 15:30:13.270446 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260202 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260206 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.260212 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.260851 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.268610 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.268628 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.269185 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.269193 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269251 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269256 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269259 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269262 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269265 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269268 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 15:30:13.270949 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269270 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269273 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269276 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269278 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269281 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269284 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269286 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269289 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269291 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269294 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269297 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269299 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269302 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269305 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269308 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269310 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269314 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269318 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269322 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 15:30:13.271313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269325 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269328 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269331 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269334 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269338 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269341 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269344 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269347 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269350 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269352 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269355 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269358 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269361 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269363 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269366 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269369 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269371 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269374 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269377 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 15:30:13.271787 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269379 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269382 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269384 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269387 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269390 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269392 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269395 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269398 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269401 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269404 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269407 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269411 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269414 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269417 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269419 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269422 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269425 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269428 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269432 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269435 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 15:30:13.272342 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269438 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269440 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269443 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269445 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269448 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269451 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269453 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269456 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269458 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269461 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269463 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269466 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269468 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269471 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269474 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269479 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269483 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269486 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269488 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269491 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 15:30:13.272843 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269494 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269496 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269499 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269502 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269505 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269507 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269510 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269512 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269515 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269518 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269521 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269524 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269527 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269530 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269533 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 15:30:13.273354 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.269538 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269666 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269671 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269675 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269677 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269680 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269683 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269685 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269688 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269690 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269693 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269696 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269698 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269701 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269703 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269706 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269708 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269711 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269714 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269716 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 15:30:13.273721 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269719 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269721 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269724 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269726 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269729 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269732 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269735 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269738 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269740 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269745 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269748 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269751 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269753 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269756 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269759 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269761 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269764 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269766 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269769 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 15:30:13.274222 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269771 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269774 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269776 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269779 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269781 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269784 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269786 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269789 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269792 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269794 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269797 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269799 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269802 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269804 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269807 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269809 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269812 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269815 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269817 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269820 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 15:30:13.274689 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269823 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269825 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269829 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269833 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269836 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269839 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269842 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269844 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269847 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269849 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269852 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269854 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269857 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269859 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269862 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269865 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269867 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269871 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269874 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 15:30:13.275191 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269877 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269880 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269883 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269886 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269888 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269891 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269894 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269896 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269899 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269901 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269904 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269907 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269925 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269928 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269931 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269934 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269936 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269940 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269944 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 15:30:13.275660 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269948 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 15:30:13.276139 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269951 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 15:30:13.276139 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:13.269954 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 15:30:13.276139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.269959 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 15:30:13.276139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.270089 2581 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 15:30:13.276139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.271982 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 15:30:13.276139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.272827 2581 server.go:1019] "Starting client certificate rotation" Sep 17 15:30:13.276139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.272940 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 15:30:13.276139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.273602 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 15:30:13.276139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.275075 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 15:30:13.279140 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.279126 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 15:30:13.287360 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.287340 2581 log.go:25] "Validated CRI v1 runtime API" Sep 17 15:30:13.290852 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.290838 2581 log.go:25] "Validated CRI v1 image API" Sep 17 15:30:13.291617 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.291596 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 15:30:13.294090 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.294070 2581 fs.go:135] Filesystem UUIDs: map[635392d9-ee1a-42b4-b4c7-578c1fb29db6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 daace8dc-f109-4eb1-a553-bcce1cdd776d:/dev/nvme0n1p4] Sep 17 15:30:13.294175 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.294088 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 15:30:13.298270 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.298162 2581 manager.go:216] Machine: {Timestamp:2026-09-17 15:30:13.297675776 +0000 UTC m=+0.293912512 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092701 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23ad29d469bef705250d5b5c2c4894 SystemUUID:ec23ad29-d469-bef7-0525-0d5b5c2c4894 BootID:0b189980-3169-4191-9398-e4c951abe829 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e9:c4:b3:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e9:c4:b3:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:66:29:96:c7:c2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 15:30:13.298270 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.298264 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 15:30:13.298403 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.298341 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 15:30:13.299560 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.299537 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 15:30:13.301058 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.301028 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 15:30:13.301189 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.301059 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-15.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 15:30:13.301663 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.301653 2581 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 15:30:13.301695 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.301665 2581 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 15:30:13.301695 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.301673 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 15:30:13.302234 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.302223 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 15:30:13.302975 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.302965 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 15:30:13.305381 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.305369 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 17 15:30:13.305497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.305488 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 15:30:13.305534 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.305516 2581 kubelet.go:486] "Attempting to sync node with API server" Sep 17 15:30:13.305534 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.305527 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 15:30:13.305591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.305540 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 15:30:13.305591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.305550 2581 kubelet.go:398] "Adding apiserver pod source" Sep 17 15:30:13.305591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.305558 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 15:30:13.305895 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.305875 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 15:30:13.305895 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.305893 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 15:30:13.307904 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.307886 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 17 15:30:13.310175 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.310161 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 15:30:13.310223 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.310197 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 15:30:13.311555 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311543 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 15:30:13.311603 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311559 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 15:30:13.311603 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311566 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 15:30:13.311603 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311574 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 15:30:13.311603 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311586 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 15:30:13.311603 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311595 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 15:30:13.311730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311609 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 15:30:13.311730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311615 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 15:30:13.311730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311621 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 15:30:13.311730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311626 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 15:30:13.311730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311640 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 15:30:13.311730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311645 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 15:30:13.311730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311669 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 15:30:13.311730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.311677 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 15:30:13.315396 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.315237 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 15:30:13.315474 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.315285 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 15:30:13.315474 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.315318 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-15.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 15:30:13.315474 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.315465 2581 server.go:1268] "Started kubelet" Sep 17 15:30:13.316208 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.316177 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 15:30:13.316284 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.316203 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 15:30:13.316284 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.316233 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 15:30:13.316928 ip-10-0-128-15 systemd[1]: Started Kubernetes Kubelet. Sep 17 15:30:13.318064 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.318046 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 15:30:13.318612 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.318589 2581 server.go:310] "Adding debug handlers to kubelet server" Sep 17 15:30:13.321433 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.321412 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 15:30:13.324394 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.324379 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 15:30:13.324861 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.324845 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 15:30:13.324946 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.324150 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-15.ec2.internal.18d62563fdbe4127 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-15.ec2.internal,UID:ip-10-0-128-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-15.ec2.internal,},FirstTimestamp:2026-09-17 15:30:13.315445031 +0000 UTC m=+0.311681767,LastTimestamp:2026-09-17 15:30:13.315445031 +0000 UTC m=+0.311681767,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-15.ec2.internal,}" Sep 17 15:30:13.324946 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.324935 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-15.ec2.internal\" not found" Sep 17 15:30:13.325488 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.325472 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 15:30:13.325586 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.325493 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 15:30:13.325586 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.325510 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 15:30:13.325698 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.325655 2581 factory.go:55] Registering systemd factory Sep 17 15:30:13.325698 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.325671 2581 factory.go:223] Registration of the systemd container factory successfully Sep 17 15:30:13.325698 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.325687 2581 reconstruct.go:97] "Volume reconstruction finished" Sep 17 15:30:13.325698 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.325696 2581 reconciler.go:29] "Reconciler: start to sync state" Sep 17 15:30:13.325891 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.325881 2581 factory.go:153] Registering CRI-O factory Sep 17 15:30:13.325964 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.325898 2581 factory.go:223] Registration of the crio container factory successfully Sep 17 15:30:13.326019 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.325966 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 15:30:13.326019 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.325985 2581 factory.go:103] Registering Raw factory Sep 17 15:30:13.326019 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.326001 2581 manager.go:1196] Started watching for new ooms in manager Sep 17 15:30:13.326311 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.326300 2581 manager.go:349] Starting recovery of all containers Sep 17 15:30:13.334976 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.334945 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vq4z8" Sep 17 15:30:13.335069 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.335048 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 15:30:13.335545 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.335531 2581 manager.go:354] Recovery completed Sep 17 15:30:13.336264 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.336237 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 15:30:13.340238 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.340221 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vq4z8" Sep 17 15:30:13.340944 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.340930 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 15:30:13.343003 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.342984 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientMemory" Sep 17 15:30:13.343073 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.343018 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 15:30:13.343073 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.343033 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientPID" Sep 17 15:30:13.343629 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.343609 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 15:30:13.343629 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.343627 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 15:30:13.343778 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.343637 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 15:30:13.343778 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.343645 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 17 15:30:13.345464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.344392 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientMemory" Sep 17 15:30:13.345464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.344413 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 15:30:13.345464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.344425 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientPID" Sep 17 15:30:13.345464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.344647 2581 policy_none.go:49] "None policy: Start" Sep 17 15:30:13.345464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.344660 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 15:30:13.345464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.344670 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 15:30:13.345773 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.345682 2581 policy_none.go:47] "Start" Sep 17 15:30:13.371602 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.371587 2581 manager.go:337] "Starting Device Plugin manager" Sep 17 15:30:13.384472 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.371630 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 15:30:13.384472 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.371643 2581 server.go:87] "Starting device plugin registration server" Sep 17 15:30:13.384472 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.371972 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 15:30:13.384472 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.371984 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 15:30:13.384472 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.372087 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 15:30:13.384472 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.372163 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 15:30:13.384472 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.372172 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 15:30:13.384472 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.372787 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 15:30:13.384472 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.372836 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-15.ec2.internal\" not found" Sep 17 15:30:13.454247 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.454221 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 15:30:13.455372 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.455353 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 15:30:13.455464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.455399 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 15:30:13.455464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.455419 2581 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 15:30:13.455464 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.455456 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 15:30:13.457736 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.457717 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 15:30:13.472782 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.472762 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 15:30:13.473687 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.473667 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientMemory" Sep 17 15:30:13.473781 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.473705 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 15:30:13.473781 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.473718 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientPID" Sep 17 15:30:13.473781 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.473739 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.481396 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.481381 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.481450 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.481406 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-15.ec2.internal\": node \"ip-10-0-128-15.ec2.internal\" not found" Sep 17 15:30:13.496712 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.496642 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-15.ec2.internal\" not found" Sep 17 15:30:13.556149 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.556117 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-15.ec2.internal"] Sep 17 15:30:13.556221 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.556190 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 15:30:13.557574 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.557560 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientMemory" Sep 17 15:30:13.557655 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.557587 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 15:30:13.557655 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.557602 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientPID" Sep 17 15:30:13.558810 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.558798 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 15:30:13.558985 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.558972 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.559028 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.558998 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 15:30:13.559548 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.559531 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientMemory" Sep 17 15:30:13.559622 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.559556 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 15:30:13.559622 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.559565 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientPID" Sep 17 15:30:13.559622 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.559593 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientMemory" Sep 17 15:30:13.559622 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.559618 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 15:30:13.559801 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.559632 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientPID" Sep 17 15:30:13.560737 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.560721 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.560817 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.560749 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 15:30:13.561394 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.561375 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientMemory" Sep 17 15:30:13.561482 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.561404 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 15:30:13.561482 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.561418 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeHasSufficientPID" Sep 17 15:30:13.587515 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.587498 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-15.ec2.internal\" not found" node="ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.591892 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.591877 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-15.ec2.internal\" not found" node="ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.596946 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.596933 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-15.ec2.internal\" not found" Sep 17 15:30:13.627404 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.627385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee901a15dc913f04af8abf50b96b53a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal\" (UID: \"6ee901a15dc913f04af8abf50b96b53a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.627496 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.627410 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e96dc0286b5d333a4741dcc145860b91-config\") pod \"kube-apiserver-proxy-ip-10-0-128-15.ec2.internal\" (UID: \"e96dc0286b5d333a4741dcc145860b91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.627496 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.627427 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ee901a15dc913f04af8abf50b96b53a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal\" (UID: \"6ee901a15dc913f04af8abf50b96b53a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.697809 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.697791 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-15.ec2.internal\" not found" Sep 17 15:30:13.728367 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.728346 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e96dc0286b5d333a4741dcc145860b91-config\") pod \"kube-apiserver-proxy-ip-10-0-128-15.ec2.internal\" (UID: \"e96dc0286b5d333a4741dcc145860b91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.728434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.728383 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e96dc0286b5d333a4741dcc145860b91-config\") pod \"kube-apiserver-proxy-ip-10-0-128-15.ec2.internal\" (UID: \"e96dc0286b5d333a4741dcc145860b91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.728434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.728416 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ee901a15dc913f04af8abf50b96b53a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal\" (UID: \"6ee901a15dc913f04af8abf50b96b53a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.728434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.728385 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6ee901a15dc913f04af8abf50b96b53a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal\" (UID: \"6ee901a15dc913f04af8abf50b96b53a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.728526 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.728446 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee901a15dc913f04af8abf50b96b53a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal\" (UID: \"6ee901a15dc913f04af8abf50b96b53a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.728526 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.728479 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee901a15dc913f04af8abf50b96b53a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal\" (UID: \"6ee901a15dc913f04af8abf50b96b53a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.798615 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.798567 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-15.ec2.internal\" not found" Sep 17 15:30:13.889189 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.889160 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.894980 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:13.894965 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-15.ec2.internal" Sep 17 15:30:13.899620 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:13.899603 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-15.ec2.internal\" not found" Sep 17 15:30:14.000089 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:14.000060 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-15.ec2.internal\" not found" Sep 17 15:30:14.101267 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:14.101233 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-15.ec2.internal\" not found" Sep 17 15:30:14.201999 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:14.201966 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-15.ec2.internal\" not found" Sep 17 15:30:14.274609 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.274579 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 15:30:14.275088 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.274734 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 15:30:14.303129 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:14.303067 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-15.ec2.internal\" not found" Sep 17 15:30:14.343998 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.343949 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:25:13 +0000 UTC" deadline="2028-06-21 03:13:21.28734453 +0000 UTC" Sep 17 15:30:14.343998 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.343978 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15419h43m6.943369856s" Sep 17 15:30:14.403849 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:14.403822 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-15.ec2.internal\" not found" Sep 17 15:30:14.404567 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.404551 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 15:30:14.425638 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.425495 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal" Sep 17 15:30:14.432339 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:14.432301 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee901a15dc913f04af8abf50b96b53a.slice/crio-78ad2460acb37352a5d23a52918bcd9ff5aff293725751e61f8f42721dd16707 WatchSource:0}: Error finding container 78ad2460acb37352a5d23a52918bcd9ff5aff293725751e61f8f42721dd16707: Status 404 returned error can't find the container with id 78ad2460acb37352a5d23a52918bcd9ff5aff293725751e61f8f42721dd16707 Sep 17 15:30:14.432655 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:14.432626 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96dc0286b5d333a4741dcc145860b91.slice/crio-1fc7100b1ec1473113c87813282a9ae4d7283631607565de0a0cfd1cb587493e WatchSource:0}: Error finding container 1fc7100b1ec1473113c87813282a9ae4d7283631607565de0a0cfd1cb587493e: Status 404 returned error can't find the container with id 1fc7100b1ec1473113c87813282a9ae4d7283631607565de0a0cfd1cb587493e Sep 17 15:30:14.436310 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.436292 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 15:30:14.438058 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.438042 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 15:30:14.440505 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.440481 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-15.ec2.internal" Sep 17 15:30:14.448844 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.448826 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 15:30:14.458845 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.458806 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal" event={"ID":"6ee901a15dc913f04af8abf50b96b53a","Type":"ContainerStarted","Data":"78ad2460acb37352a5d23a52918bcd9ff5aff293725751e61f8f42721dd16707"} Sep 17 15:30:14.460086 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.460065 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-15.ec2.internal" event={"ID":"e96dc0286b5d333a4741dcc145860b91","Type":"ContainerStarted","Data":"1fc7100b1ec1473113c87813282a9ae4d7283631607565de0a0cfd1cb587493e"} Sep 17 15:30:14.560319 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.560202 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 15:30:14.878205 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:14.878132 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 15:30:15.128339 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.128252 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 15:30:15.307464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.307425 2581 apiserver.go:52] "Watching apiserver" Sep 17 15:30:15.314204 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.314177 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 15:30:15.315408 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.315383 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-15.ec2.internal","openshift-cluster-node-tuning-operator/tuned-x2wwn","openshift-image-registry/node-ca-d7hx7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal","openshift-multus/multus-5r65k","openshift-multus/multus-additional-cni-plugins-lsxcg","openshift-network-operator/iptables-alerter-44mlh","kube-system/global-pull-secret-syncer-5rf6m","kube-system/konnectivity-agent-lv4s5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp","openshift-dns/node-resolver-wd878","openshift-multus/network-metrics-daemon-cgcpl","openshift-network-diagnostics/network-check-target-6mlp5","openshift-ovn-kubernetes/ovnkube-node-cvwgv"] Sep 17 15:30:15.316974 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.316954 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.318700 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.318677 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d7hx7" Sep 17 15:30:15.319973 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.319708 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 15:30:15.319973 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.319721 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 15:30:15.319973 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.319814 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n48m4\"" Sep 17 15:30:15.321892 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.321081 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 15:30:15.321892 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.321377 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 15:30:15.321892 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.321572 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 15:30:15.321892 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.321752 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dclrc\"" Sep 17 15:30:15.323163 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.323068 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.324561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.324542 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.324820 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.324800 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 15:30:15.325044 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.325026 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 15:30:15.325247 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.325231 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7fpxx\"" Sep 17 15:30:15.325784 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.325474 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 15:30:15.325784 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.325613 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 15:30:15.325784 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.325616 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 15:30:15.325784 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.325755 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-44mlh" Sep 17 15:30:15.326652 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.326632 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 15:30:15.326757 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.326670 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 15:30:15.326837 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.326815 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f75s5\"" Sep 17 15:30:15.327065 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.327045 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:15.327165 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:15.327140 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:15.327700 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.327679 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 15:30:15.328124 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.328105 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 15:30:15.328421 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.328401 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 15:30:15.328635 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.328616 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sncnc\"" Sep 17 15:30:15.330425 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.330407 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lv4s5" Sep 17 15:30:15.332426 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.331884 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.332523 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.332469 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 15:30:15.333243 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.333221 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8dwgb\"" Sep 17 15:30:15.335362 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.335119 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 15:30:15.336521 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.336309 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 15:30:15.336616 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.336603 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q562k\"" Sep 17 15:30:15.338844 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.337108 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 15:30:15.338844 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.337334 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 15:30:15.338844 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.337362 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:15.338844 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:15.337447 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:15.338844 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.337572 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wd878" Sep 17 15:30:15.338844 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.338628 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:15.338844 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:15.338724 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.340072 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.340692 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.340919 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j9xbm\"" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341105 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-lib-modules\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341156 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-host\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341194 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-system-cni-dir\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341229 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-sysconfig\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341259 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-sys\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341327 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e716979-d391-4561-b731-64cd1b6efed3-etc-tuned\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341398 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0402d124-60fa-4a15-86cc-cac6e6904bb3-serviceca\") pod \"node-ca-d7hx7\" (UID: \"0402d124-60fa-4a15-86cc-cac6e6904bb3\") " pod="openshift-image-registry/node-ca-d7hx7" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341432 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-cnibin\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341470 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-hostroot\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341501 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-etc-kubernetes\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341533 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/899f7ef8-d770-4340-bd86-7552cd1ac461-cnibin\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341589 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-modprobe-d\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-multus-socket-dir-parent\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341656 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-var-lib-cni-multus\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-var-lib-kubelet\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.341876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341719 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/680c1df0-298c-464a-a1cf-4739d24ba0fa-multus-daemon-config\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341751 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/899f7ef8-d770-4340-bd86-7552cd1ac461-cni-binary-copy\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341810 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/899f7ef8-d770-4340-bd86-7552cd1ac461-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-run-k8s-cni-cncf-io\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.341974 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/899f7ef8-d770-4340-bd86-7552cd1ac461-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342015 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkt4v\" (UniqueName: \"kubernetes.io/projected/899f7ef8-d770-4340-bd86-7552cd1ac461-kube-api-access-nkt4v\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342060 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-multus-cni-dir\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342096 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-run-multus-certs\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342133 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/721e805e-1d0a-473c-be1a-871d62cfebbe-kubelet-config\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342168 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/721e805e-1d0a-473c-be1a-871d62cfebbe-dbus\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342211 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-sysctl-conf\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342246 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/680c1df0-298c-464a-a1cf-4739d24ba0fa-cni-binary-copy\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342279 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/899f7ef8-d770-4340-bd86-7552cd1ac461-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342316 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f68347c-394c-4828-b962-ffe2684471cb-iptables-alerter-script\") pod \"iptables-alerter-44mlh\" (UID: \"9f68347c-394c-4828-b962-ffe2684471cb\") " pod="openshift-network-operator/iptables-alerter-44mlh" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342413 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342457 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-sysctl-d\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.342878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342523 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-systemd\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342559 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-run\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342593 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e716979-d391-4561-b731-64cd1b6efed3-tmp\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342624 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0402d124-60fa-4a15-86cc-cac6e6904bb3-host\") pod \"node-ca-d7hx7\" (UID: \"0402d124-60fa-4a15-86cc-cac6e6904bb3\") " pod="openshift-image-registry/node-ca-d7hx7" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342653 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7st2\" (UniqueName: \"kubernetes.io/projected/0402d124-60fa-4a15-86cc-cac6e6904bb3-kube-api-access-g7st2\") pod \"node-ca-d7hx7\" (UID: \"0402d124-60fa-4a15-86cc-cac6e6904bb3\") " pod="openshift-image-registry/node-ca-d7hx7" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342685 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-var-lib-kubelet\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342716 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6h88\" (UniqueName: \"kubernetes.io/projected/3e716979-d391-4561-b731-64cd1b6efed3-kube-api-access-p6h88\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-run-netns\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342789 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-var-lib-cni-bin\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342821 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-multus-conf-dir\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342853 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s5tw\" (UniqueName: \"kubernetes.io/projected/680c1df0-298c-464a-a1cf-4739d24ba0fa-kube-api-access-8s5tw\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342891 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/899f7ef8-d770-4340-bd86-7552cd1ac461-system-cni-dir\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342925 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f68347c-394c-4828-b962-ffe2684471cb-host-slash\") pod \"iptables-alerter-44mlh\" (UID: \"9f68347c-394c-4828-b962-ffe2684471cb\") " pod="openshift-network-operator/iptables-alerter-44mlh" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g87jb\" (UniqueName: \"kubernetes.io/projected/9f68347c-394c-4828-b962-ffe2684471cb-kube-api-access-g87jb\") pod \"iptables-alerter-44mlh\" (UID: \"9f68347c-394c-4828-b962-ffe2684471cb\") " pod="openshift-network-operator/iptables-alerter-44mlh" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.342987 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-kubernetes\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.343342 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.343604 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 15:30:15.343673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.343655 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gz2nh\"" Sep 17 15:30:15.344512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.343823 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 15:30:15.345068 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.345050 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 15:30:15.345447 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.345426 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 15:30:15.345564 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.345540 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 15:30:15.345889 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.345870 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 15:30:15.346222 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.346117 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 15:30:15.363920 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.363895 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p76bz" Sep 17 15:30:15.370525 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.370504 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p76bz" Sep 17 15:30:15.427018 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.426909 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 15:30:15.443450 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443423 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-node-log\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.443450 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443453 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/573f3972-eb62-40d9-aff1-c1baf1d15362-ovnkube-config\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.443681 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443472 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5tzd\" (UniqueName: \"kubernetes.io/projected/fa257b5d-59d6-4286-a99f-b2bee16de288-kube-api-access-v5tzd\") pod \"network-metrics-daemon-cgcpl\" (UID: \"fa257b5d-59d6-4286-a99f-b2bee16de288\") " pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:15.443681 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443496 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-modprobe-d\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.443681 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443529 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-multus-socket-dir-parent\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.443681 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-var-lib-cni-multus\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.443681 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443631 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-var-lib-cni-multus\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.443681 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443667 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-var-lib-kubelet\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.443681 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443665 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-modprobe-d\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443696 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8fzq\" (UniqueName: \"kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq\") pod \"network-check-target-6mlp5\" (UID: \"af87ecf6-e9d9-4c65-ace2-bc1709237610\") " pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443720 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-multus-socket-dir-parent\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443733 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-var-lib-kubelet\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443735 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443796 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qv8\" (UniqueName: \"kubernetes.io/projected/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-kube-api-access-w7qv8\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443830 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-run-k8s-cni-cncf-io\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443856 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18b5b4bc-a399-468b-9957-2d51f1f24c6c-agent-certs\") pod \"konnectivity-agent-lv4s5\" (UID: \"18b5b4bc-a399-468b-9957-2d51f1f24c6c\") " pod="kube-system/konnectivity-agent-lv4s5" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443880 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-cni-bin\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443908 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-multus-cni-dir\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443905 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-run-k8s-cni-cncf-io\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443950 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-run-multus-certs\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.443980 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-slash\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444015 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-log-socket\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444086 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-run-multus-certs\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444122 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-multus-cni-dir\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.444131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444123 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-cni-netd\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444154 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-device-dir\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444194 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/721e805e-1d0a-473c-be1a-871d62cfebbe-dbus\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444240 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-sysctl-conf\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444254 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/721e805e-1d0a-473c-be1a-871d62cfebbe-dbus\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444270 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-kubelet\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444327 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-systemd-units\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-sysctl-d\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444383 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-systemd\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444417 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-run\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444418 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-sysctl-conf\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444441 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0402d124-60fa-4a15-86cc-cac6e6904bb3-host\") pod \"node-ca-d7hx7\" (UID: \"0402d124-60fa-4a15-86cc-cac6e6904bb3\") " pod="openshift-image-registry/node-ca-d7hx7" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444466 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-multus-conf-dir\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444478 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-systemd\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444490 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s5tw\" (UniqueName: \"kubernetes.io/projected/680c1df0-298c-464a-a1cf-4739d24ba0fa-kube-api-access-8s5tw\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444516 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-run-ovn-kubernetes\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444523 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0402d124-60fa-4a15-86cc-cac6e6904bb3-host\") pod \"node-ca-d7hx7\" (UID: \"0402d124-60fa-4a15-86cc-cac6e6904bb3\") " pod="openshift-image-registry/node-ca-d7hx7" Sep 17 15:30:15.444899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444539 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-sysctl-d\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444546 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq7l4\" (UniqueName: \"kubernetes.io/projected/573f3972-eb62-40d9-aff1-c1baf1d15362-kube-api-access-zq7l4\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444567 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-run\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-var-lib-kubelet\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-run-netns\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444606 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-multus-conf-dir\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444684 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-var-lib-kubelet\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444703 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/899f7ef8-d770-4340-bd86-7552cd1ac461-system-cni-dir\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444729 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-run-netns\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444742 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18b5b4bc-a399-468b-9957-2d51f1f24c6c-konnectivity-ca\") pod \"konnectivity-agent-lv4s5\" (UID: \"18b5b4bc-a399-468b-9957-2d51f1f24c6c\") " pod="kube-system/konnectivity-agent-lv4s5" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444767 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-run-systemd\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444772 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/899f7ef8-d770-4340-bd86-7552cd1ac461-system-cni-dir\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444796 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/573f3972-eb62-40d9-aff1-c1baf1d15362-ovn-node-metrics-cert\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444845 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-sys-fs\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444869 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-host\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-system-cni-dir\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444921 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-var-lib-openvswitch\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.445770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444971 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-host\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.444981 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-socket-dir\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445059 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-cnibin\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445067 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-system-cni-dir\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-hostroot\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445126 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-cnibin\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445133 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-etc-kubernetes\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445167 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-hostroot\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445196 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/899f7ef8-d770-4340-bd86-7552cd1ac461-cnibin\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445228 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-etc-kubernetes\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445246 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/899f7ef8-d770-4340-bd86-7552cd1ac461-cni-binary-copy\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445292 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/899f7ef8-d770-4340-bd86-7552cd1ac461-cnibin\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445324 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/899f7ef8-d770-4340-bd86-7552cd1ac461-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445355 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-run-openvswitch\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445393 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/573f3972-eb62-40d9-aff1-c1baf1d15362-ovnkube-script-lib\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445419 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/680c1df0-298c-464a-a1cf-4739d24ba0fa-multus-daemon-config\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.446512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445480 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445504 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30c3098f-6b70-43ef-96fe-67a7c9f4ff91-hosts-file\") pod \"node-resolver-wd878\" (UID: \"30c3098f-6b70-43ef-96fe-67a7c9f4ff91\") " pod="openshift-dns/node-resolver-wd878" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445527 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c3098f-6b70-43ef-96fe-67a7c9f4ff91-tmp-dir\") pod \"node-resolver-wd878\" (UID: \"30c3098f-6b70-43ef-96fe-67a7c9f4ff91\") " pod="openshift-dns/node-resolver-wd878" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445555 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/899f7ef8-d770-4340-bd86-7552cd1ac461-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445590 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkt4v\" (UniqueName: \"kubernetes.io/projected/899f7ef8-d770-4340-bd86-7552cd1ac461-kube-api-access-nkt4v\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445621 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-etc-openvswitch\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445651 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-run-ovn\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445697 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/573f3972-eb62-40d9-aff1-c1baf1d15362-env-overrides\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445723 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/721e805e-1d0a-473c-be1a-871d62cfebbe-kubelet-config\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445748 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/680c1df0-298c-464a-a1cf-4739d24ba0fa-cni-binary-copy\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445775 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/899f7ef8-d770-4340-bd86-7552cd1ac461-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445803 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-registration-dir\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445833 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djjd\" (UniqueName: \"kubernetes.io/projected/30c3098f-6b70-43ef-96fe-67a7c9f4ff91-kube-api-access-8djjd\") pod \"node-resolver-wd878\" (UID: \"30c3098f-6b70-43ef-96fe-67a7c9f4ff91\") " pod="openshift-dns/node-resolver-wd878" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445862 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f68347c-394c-4828-b962-ffe2684471cb-iptables-alerter-script\") pod \"iptables-alerter-44mlh\" (UID: \"9f68347c-394c-4828-b962-ffe2684471cb\") " pod="openshift-network-operator/iptables-alerter-44mlh" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445915 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/899f7ef8-d770-4340-bd86-7552cd1ac461-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445928 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/899f7ef8-d770-4340-bd86-7552cd1ac461-cni-binary-copy\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.447418 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.445986 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/721e805e-1d0a-473c-be1a-871d62cfebbe-kubelet-config\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446179 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/899f7ef8-d770-4340-bd86-7552cd1ac461-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446495 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/680c1df0-298c-464a-a1cf-4739d24ba0fa-multus-daemon-config\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446535 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/899f7ef8-d770-4340-bd86-7552cd1ac461-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446542 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446572 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e716979-d391-4561-b731-64cd1b6efed3-tmp\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446598 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7st2\" (UniqueName: \"kubernetes.io/projected/0402d124-60fa-4a15-86cc-cac6e6904bb3-kube-api-access-g7st2\") pod \"node-ca-d7hx7\" (UID: \"0402d124-60fa-4a15-86cc-cac6e6904bb3\") " pod="openshift-image-registry/node-ca-d7hx7" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446621 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-var-lib-cni-bin\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446655 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-run-netns\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:15.446662 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446683 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6h88\" (UniqueName: \"kubernetes.io/projected/3e716979-d391-4561-b731-64cd1b6efed3-kube-api-access-p6h88\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:15.446751 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret podName:721e805e-1d0a-473c-be1a-871d62cfebbe nodeName:}" failed. No retries permitted until 2026-09-17 15:30:15.946712978 +0000 UTC m=+2.942949724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret") pod "global-pull-secret-syncer-5rf6m" (UID: "721e805e-1d0a-473c-be1a-871d62cfebbe") : object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446776 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f68347c-394c-4828-b962-ffe2684471cb-host-slash\") pod \"iptables-alerter-44mlh\" (UID: \"9f68347c-394c-4828-b962-ffe2684471cb\") " pod="openshift-network-operator/iptables-alerter-44mlh" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446785 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/680c1df0-298c-464a-a1cf-4739d24ba0fa-cni-binary-copy\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446809 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g87jb\" (UniqueName: \"kubernetes.io/projected/9f68347c-394c-4828-b962-ffe2684471cb-kube-api-access-g87jb\") pod \"iptables-alerter-44mlh\" (UID: \"9f68347c-394c-4828-b962-ffe2684471cb\") " pod="openshift-network-operator/iptables-alerter-44mlh" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446843 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/680c1df0-298c-464a-a1cf-4739d24ba0fa-host-var-lib-cni-bin\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446835 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-kubernetes\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.448139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446876 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-lib-modules\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.448922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446940 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f68347c-394c-4828-b962-ffe2684471cb-iptables-alerter-script\") pod \"iptables-alerter-44mlh\" (UID: \"9f68347c-394c-4828-b962-ffe2684471cb\") " pod="openshift-network-operator/iptables-alerter-44mlh" Sep 17 15:30:15.448922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.446940 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f68347c-394c-4828-b962-ffe2684471cb-host-slash\") pod \"iptables-alerter-44mlh\" (UID: \"9f68347c-394c-4828-b962-ffe2684471cb\") " pod="openshift-network-operator/iptables-alerter-44mlh" Sep 17 15:30:15.448922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.447025 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs\") pod \"network-metrics-daemon-cgcpl\" (UID: \"fa257b5d-59d6-4286-a99f-b2bee16de288\") " pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:15.448922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.447064 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-sysconfig\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.448922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.447100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-sys\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.448922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.447075 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-lib-modules\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.448922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.447115 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-kubernetes\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.448922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.447136 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-etc-sysconfig\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.448922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.447125 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e716979-d391-4561-b731-64cd1b6efed3-etc-tuned\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.448922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.447270 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e716979-d391-4561-b731-64cd1b6efed3-sys\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.448922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.447274 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0402d124-60fa-4a15-86cc-cac6e6904bb3-serviceca\") pod \"node-ca-d7hx7\" (UID: \"0402d124-60fa-4a15-86cc-cac6e6904bb3\") " pod="openshift-image-registry/node-ca-d7hx7" Sep 17 15:30:15.448922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.447522 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 15:30:15.448922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.447689 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0402d124-60fa-4a15-86cc-cac6e6904bb3-serviceca\") pod \"node-ca-d7hx7\" (UID: \"0402d124-60fa-4a15-86cc-cac6e6904bb3\") " pod="openshift-image-registry/node-ca-d7hx7" Sep 17 15:30:15.451633 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.451603 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e716979-d391-4561-b731-64cd1b6efed3-etc-tuned\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.451767 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.451694 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e716979-d391-4561-b731-64cd1b6efed3-tmp\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.455395 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.455370 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s5tw\" (UniqueName: \"kubernetes.io/projected/680c1df0-298c-464a-a1cf-4739d24ba0fa-kube-api-access-8s5tw\") pod \"multus-5r65k\" (UID: \"680c1df0-298c-464a-a1cf-4739d24ba0fa\") " pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.456187 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.456161 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkt4v\" (UniqueName: \"kubernetes.io/projected/899f7ef8-d770-4340-bd86-7552cd1ac461-kube-api-access-nkt4v\") pod \"multus-additional-cni-plugins-lsxcg\" (UID: \"899f7ef8-d770-4340-bd86-7552cd1ac461\") " pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.464081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.464050 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6h88\" (UniqueName: \"kubernetes.io/projected/3e716979-d391-4561-b731-64cd1b6efed3-kube-api-access-p6h88\") pod \"tuned-x2wwn\" (UID: \"3e716979-d391-4561-b731-64cd1b6efed3\") " pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.464877 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.464852 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g87jb\" (UniqueName: \"kubernetes.io/projected/9f68347c-394c-4828-b962-ffe2684471cb-kube-api-access-g87jb\") pod \"iptables-alerter-44mlh\" (UID: \"9f68347c-394c-4828-b962-ffe2684471cb\") " pod="openshift-network-operator/iptables-alerter-44mlh" Sep 17 15:30:15.465009 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.464852 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7st2\" (UniqueName: \"kubernetes.io/projected/0402d124-60fa-4a15-86cc-cac6e6904bb3-kube-api-access-g7st2\") pod \"node-ca-d7hx7\" (UID: \"0402d124-60fa-4a15-86cc-cac6e6904bb3\") " pod="openshift-image-registry/node-ca-d7hx7" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547543 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fzq\" (UniqueName: \"kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq\") pod \"network-check-target-6mlp5\" (UID: \"af87ecf6-e9d9-4c65-ace2-bc1709237610\") " pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547596 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qv8\" (UniqueName: \"kubernetes.io/projected/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-kube-api-access-w7qv8\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547653 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18b5b4bc-a399-468b-9957-2d51f1f24c6c-agent-certs\") pod \"konnectivity-agent-lv4s5\" (UID: \"18b5b4bc-a399-468b-9957-2d51f1f24c6c\") " pod="kube-system/konnectivity-agent-lv4s5" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547682 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-cni-bin\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547726 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547741 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-slash\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547778 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-cni-bin\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547766 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-log-socket\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547796 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-slash\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547854 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-log-socket\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547856 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-cni-netd\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547894 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-device-dir\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547897 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-cni-netd\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-kubelet\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547946 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-device-dir\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.548001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.547983 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-kubelet\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.548921 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.548000 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-systemd-units\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.548921 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.548065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-run-ovn-kubernetes\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.548921 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.548088 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7l4\" (UniqueName: \"kubernetes.io/projected/573f3972-eb62-40d9-aff1-c1baf1d15362-kube-api-access-zq7l4\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.548921 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.548093 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-systemd-units\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.549161 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.548122 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18b5b4bc-a399-468b-9957-2d51f1f24c6c-konnectivity-ca\") pod \"konnectivity-agent-lv4s5\" (UID: \"18b5b4bc-a399-468b-9957-2d51f1f24c6c\") " pod="kube-system/konnectivity-agent-lv4s5" Sep 17 15:30:15.549223 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549172 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-run-systemd\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.549223 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549213 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/573f3972-eb62-40d9-aff1-c1baf1d15362-ovn-node-metrics-cert\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.549334 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-sys-fs\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.549334 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549326 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-run-systemd\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.549496 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549378 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-run-ovn-kubernetes\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.549496 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549420 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/18b5b4bc-a399-468b-9957-2d51f1f24c6c-konnectivity-ca\") pod \"konnectivity-agent-lv4s5\" (UID: \"18b5b4bc-a399-468b-9957-2d51f1f24c6c\") " pod="kube-system/konnectivity-agent-lv4s5" Sep 17 15:30:15.549496 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549464 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-sys-fs\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.549644 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549499 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-var-lib-openvswitch\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.549644 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549531 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-socket-dir\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.549644 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549573 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-run-openvswitch\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.549644 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/573f3972-eb62-40d9-aff1-c1baf1d15362-ovnkube-script-lib\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.549846 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549638 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-var-lib-openvswitch\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.549846 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549701 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-run-openvswitch\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.549846 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.549846 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549722 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-socket-dir\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.549846 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549748 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.549846 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549761 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.549846 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549823 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30c3098f-6b70-43ef-96fe-67a7c9f4ff91-hosts-file\") pod \"node-resolver-wd878\" (UID: \"30c3098f-6b70-43ef-96fe-67a7c9f4ff91\") " pod="openshift-dns/node-resolver-wd878" Sep 17 15:30:15.550228 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549866 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c3098f-6b70-43ef-96fe-67a7c9f4ff91-tmp-dir\") pod \"node-resolver-wd878\" (UID: \"30c3098f-6b70-43ef-96fe-67a7c9f4ff91\") " pod="openshift-dns/node-resolver-wd878" Sep 17 15:30:15.550228 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549900 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.550228 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549904 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-etc-openvswitch\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.550228 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.549987 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-etc-openvswitch\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.550228 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.550014 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-run-ovn\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.550228 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.550058 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/573f3972-eb62-40d9-aff1-c1baf1d15362-env-overrides\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.550228 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.550103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-registration-dir\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.550228 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.550139 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/30c3098f-6b70-43ef-96fe-67a7c9f4ff91-hosts-file\") pod \"node-resolver-wd878\" (UID: \"30c3098f-6b70-43ef-96fe-67a7c9f4ff91\") " pod="openshift-dns/node-resolver-wd878" Sep 17 15:30:15.550618 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.550281 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-run-ovn\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.550618 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.550414 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c3098f-6b70-43ef-96fe-67a7c9f4ff91-tmp-dir\") pod \"node-resolver-wd878\" (UID: \"30c3098f-6b70-43ef-96fe-67a7c9f4ff91\") " pod="openshift-dns/node-resolver-wd878" Sep 17 15:30:15.551002 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.550962 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/573f3972-eb62-40d9-aff1-c1baf1d15362-env-overrides\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.551073 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.551041 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/573f3972-eb62-40d9-aff1-c1baf1d15362-ovnkube-script-lib\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.551167 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djjd\" (UniqueName: \"kubernetes.io/projected/30c3098f-6b70-43ef-96fe-67a7c9f4ff91-kube-api-access-8djjd\") pod \"node-resolver-wd878\" (UID: \"30c3098f-6b70-43ef-96fe-67a7c9f4ff91\") " pod="openshift-dns/node-resolver-wd878" Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.551243 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-registration-dir\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.551257 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-run-netns\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.551318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs\") pod \"network-metrics-daemon-cgcpl\" (UID: \"fa257b5d-59d6-4286-a99f-b2bee16de288\") " pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.551370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-node-log\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.551413 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/18b5b4bc-a399-468b-9957-2d51f1f24c6c-agent-certs\") pod \"konnectivity-agent-lv4s5\" (UID: \"18b5b4bc-a399-468b-9957-2d51f1f24c6c\") " pod="kube-system/konnectivity-agent-lv4s5" Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.551497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/573f3972-eb62-40d9-aff1-c1baf1d15362-ovnkube-config\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.551529 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5tzd\" (UniqueName: \"kubernetes.io/projected/fa257b5d-59d6-4286-a99f-b2bee16de288-kube-api-access-v5tzd\") pod \"network-metrics-daemon-cgcpl\" (UID: \"fa257b5d-59d6-4286-a99f-b2bee16de288\") " pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:15.551606 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.551634 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-node-log\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:15.551683 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs podName:fa257b5d-59d6-4286-a99f-b2bee16de288 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:16.051661688 +0000 UTC m=+3.047898432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs") pod "network-metrics-daemon-cgcpl" (UID: "fa257b5d-59d6-4286-a99f-b2bee16de288") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.551692 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/573f3972-eb62-40d9-aff1-c1baf1d15362-host-run-netns\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.551988 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/573f3972-eb62-40d9-aff1-c1baf1d15362-ovn-node-metrics-cert\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.554081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.552312 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/573f3972-eb62-40d9-aff1-c1baf1d15362-ovnkube-config\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.560619 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:15.560594 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 15:30:15.560704 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:15.560624 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 15:30:15.560704 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:15.560640 2581 projected.go:196] Error preparing data for projected volume kube-api-access-h8fzq for pod openshift-network-diagnostics/network-check-target-6mlp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:15.560823 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:15.560709 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq podName:af87ecf6-e9d9-4c65-ace2-bc1709237610 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:16.060691218 +0000 UTC m=+3.056927968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h8fzq" (UniqueName: "kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq") pod "network-check-target-6mlp5" (UID: "af87ecf6-e9d9-4c65-ace2-bc1709237610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:15.563185 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.563156 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djjd\" (UniqueName: \"kubernetes.io/projected/30c3098f-6b70-43ef-96fe-67a7c9f4ff91-kube-api-access-8djjd\") pod \"node-resolver-wd878\" (UID: \"30c3098f-6b70-43ef-96fe-67a7c9f4ff91\") " pod="openshift-dns/node-resolver-wd878" Sep 17 15:30:15.563287 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.563265 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qv8\" (UniqueName: \"kubernetes.io/projected/07ffcf0a-8dd5-4289-8c98-b1bbb48270ae-kube-api-access-w7qv8\") pod \"aws-ebs-csi-driver-node-6mhqp\" (UID: \"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.563351 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.563323 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5tzd\" (UniqueName: \"kubernetes.io/projected/fa257b5d-59d6-4286-a99f-b2bee16de288-kube-api-access-v5tzd\") pod \"network-metrics-daemon-cgcpl\" (UID: \"fa257b5d-59d6-4286-a99f-b2bee16de288\") " pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:15.563460 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.563440 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq7l4\" (UniqueName: \"kubernetes.io/projected/573f3972-eb62-40d9-aff1-c1baf1d15362-kube-api-access-zq7l4\") pod \"ovnkube-node-cvwgv\" (UID: \"573f3972-eb62-40d9-aff1-c1baf1d15362\") " pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.633955 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.633868 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" Sep 17 15:30:15.643693 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.643665 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d7hx7" Sep 17 15:30:15.655701 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.655674 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5r65k" Sep 17 15:30:15.663436 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.663416 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lsxcg" Sep 17 15:30:15.672217 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.672194 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-44mlh" Sep 17 15:30:15.681090 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.681071 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lv4s5" Sep 17 15:30:15.686325 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.686304 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" Sep 17 15:30:15.695030 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.695013 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wd878" Sep 17 15:30:15.701719 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.701690 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:15.955063 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:15.954976 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:15.955228 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:15.955132 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:15.955228 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:15.955210 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret podName:721e805e-1d0a-473c-be1a-871d62cfebbe nodeName:}" failed. No retries permitted until 2026-09-17 15:30:16.955187557 +0000 UTC m=+3.951424308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret") pod "global-pull-secret-syncer-5rf6m" (UID: "721e805e-1d0a-473c-be1a-871d62cfebbe") : object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:16.056219 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:16.056177 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs\") pod \"network-metrics-daemon-cgcpl\" (UID: \"fa257b5d-59d6-4286-a99f-b2bee16de288\") " pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:16.056410 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:16.056364 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:16.056463 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:16.056446 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs podName:fa257b5d-59d6-4286-a99f-b2bee16de288 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:17.056424304 +0000 UTC m=+4.052661045 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs") pod "network-metrics-daemon-cgcpl" (UID: "fa257b5d-59d6-4286-a99f-b2bee16de288") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:16.157320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:16.157282 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fzq\" (UniqueName: \"kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq\") pod \"network-check-target-6mlp5\" (UID: \"af87ecf6-e9d9-4c65-ace2-bc1709237610\") " pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:16.157499 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:16.157452 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 15:30:16.157499 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:16.157472 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 15:30:16.157499 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:16.157485 2581 projected.go:196] Error preparing data for projected volume kube-api-access-h8fzq for pod openshift-network-diagnostics/network-check-target-6mlp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:16.157670 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:16.157555 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq podName:af87ecf6-e9d9-4c65-ace2-bc1709237610 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:17.157535709 +0000 UTC m=+4.153772456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h8fzq" (UniqueName: "kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq") pod "network-check-target-6mlp5" (UID: "af87ecf6-e9d9-4c65-ace2-bc1709237610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:16.373694 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:16.373601 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:25:15 +0000 UTC" deadline="2028-04-05 21:30:38.967623332 +0000 UTC" Sep 17 15:30:16.373694 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:16.373640 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13590h0m22.593987394s" Sep 17 15:30:16.496246 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:16.496217 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899f7ef8_d770_4340_bd86_7552cd1ac461.slice/crio-970ba94ae4f50bb8b9d5a2564c2c3ada5094f97b570292acda28c195729936e5 WatchSource:0}: Error finding container 970ba94ae4f50bb8b9d5a2564c2c3ada5094f97b570292acda28c195729936e5: Status 404 returned error can't find the container with id 970ba94ae4f50bb8b9d5a2564c2c3ada5094f97b570292acda28c195729936e5 Sep 17 15:30:16.498344 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:16.498319 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f68347c_394c_4828_b962_ffe2684471cb.slice/crio-317a268b736d14cf228657078904211bb9a30fe106153ea4856303b0e06b4d31 WatchSource:0}: Error finding container 317a268b736d14cf228657078904211bb9a30fe106153ea4856303b0e06b4d31: Status 404 returned error can't find the container with id 317a268b736d14cf228657078904211bb9a30fe106153ea4856303b0e06b4d31 Sep 17 15:30:16.502641 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:16.502113 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573f3972_eb62_40d9_aff1_c1baf1d15362.slice/crio-9c0202765b7b6fb7a78db486d7d5bac61ac0f3d827ff5cafc43509861822c1d5 WatchSource:0}: Error finding container 9c0202765b7b6fb7a78db486d7d5bac61ac0f3d827ff5cafc43509861822c1d5: Status 404 returned error can't find the container with id 9c0202765b7b6fb7a78db486d7d5bac61ac0f3d827ff5cafc43509861822c1d5 Sep 17 15:30:16.502909 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:16.502851 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b5b4bc_a399_468b_9957_2d51f1f24c6c.slice/crio-23bd5c7447e09262a41a358b1a514ef7ef20dfc0f6aed4ef7e618d24654f6289 WatchSource:0}: Error finding container 23bd5c7447e09262a41a358b1a514ef7ef20dfc0f6aed4ef7e618d24654f6289: Status 404 returned error can't find the container with id 23bd5c7447e09262a41a358b1a514ef7ef20dfc0f6aed4ef7e618d24654f6289 Sep 17 15:30:16.504736 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:16.504712 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680c1df0_298c_464a_a1cf_4739d24ba0fa.slice/crio-148672b45e229931fea73f80afe822973db1e68f56596d8ad0fd0b11209a314e WatchSource:0}: Error finding container 148672b45e229931fea73f80afe822973db1e68f56596d8ad0fd0b11209a314e: Status 404 returned error can't find the container with id 148672b45e229931fea73f80afe822973db1e68f56596d8ad0fd0b11209a314e Sep 17 15:30:16.507089 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:16.506560 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e716979_d391_4561_b731_64cd1b6efed3.slice/crio-23acb8ace595529e3e020089a4ecff77d596881d6d143f944ae38305a33bac15 WatchSource:0}: Error finding container 23acb8ace595529e3e020089a4ecff77d596881d6d143f944ae38305a33bac15: Status 404 returned error can't find the container with id 23acb8ace595529e3e020089a4ecff77d596881d6d143f944ae38305a33bac15 Sep 17 15:30:16.507089 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:16.506884 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0402d124_60fa_4a15_86cc_cac6e6904bb3.slice/crio-d5262136ad2dc389b674349dde83a62161cad7677709fd133b8dfd2f52db0c54 WatchSource:0}: Error finding container d5262136ad2dc389b674349dde83a62161cad7677709fd133b8dfd2f52db0c54: Status 404 returned error can't find the container with id d5262136ad2dc389b674349dde83a62161cad7677709fd133b8dfd2f52db0c54 Sep 17 15:30:16.507443 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:16.507122 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ffcf0a_8dd5_4289_8c98_b1bbb48270ae.slice/crio-4a5d131e8fb3a4d995f8aae7fd0366664a31fba50b4719adaa97591ae9b873d6 WatchSource:0}: Error finding container 4a5d131e8fb3a4d995f8aae7fd0366664a31fba50b4719adaa97591ae9b873d6: Status 404 returned error can't find the container with id 4a5d131e8fb3a4d995f8aae7fd0366664a31fba50b4719adaa97591ae9b873d6 Sep 17 15:30:16.509753 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:16.509552 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c3098f_6b70_43ef_96fe_67a7c9f4ff91.slice/crio-e1f592f6a83805c5c9476e19c3855616fdb1e7ce41a2c154c94a5a261aa465bc WatchSource:0}: Error finding container e1f592f6a83805c5c9476e19c3855616fdb1e7ce41a2c154c94a5a261aa465bc: Status 404 returned error can't find the container with id e1f592f6a83805c5c9476e19c3855616fdb1e7ce41a2c154c94a5a261aa465bc Sep 17 15:30:16.964690 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:16.964602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:16.964863 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:16.964829 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:16.964925 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:16.964899 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret podName:721e805e-1d0a-473c-be1a-871d62cfebbe nodeName:}" failed. No retries permitted until 2026-09-17 15:30:18.964878425 +0000 UTC m=+5.961115167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret") pod "global-pull-secret-syncer-5rf6m" (UID: "721e805e-1d0a-473c-be1a-871d62cfebbe") : object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:17.066125 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.065489 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs\") pod \"network-metrics-daemon-cgcpl\" (UID: \"fa257b5d-59d6-4286-a99f-b2bee16de288\") " pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:17.066125 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:17.065656 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:17.066125 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:17.065726 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs podName:fa257b5d-59d6-4286-a99f-b2bee16de288 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:19.06570449 +0000 UTC m=+6.061941232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs") pod "network-metrics-daemon-cgcpl" (UID: "fa257b5d-59d6-4286-a99f-b2bee16de288") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:17.167787 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.167135 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fzq\" (UniqueName: \"kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq\") pod \"network-check-target-6mlp5\" (UID: \"af87ecf6-e9d9-4c65-ace2-bc1709237610\") " pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:17.167787 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:17.167327 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 15:30:17.167787 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:17.167351 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 15:30:17.167787 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:17.167366 2581 projected.go:196] Error preparing data for projected volume kube-api-access-h8fzq for pod openshift-network-diagnostics/network-check-target-6mlp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:17.167787 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:17.167427 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq podName:af87ecf6-e9d9-4c65-ace2-bc1709237610 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:19.167406522 +0000 UTC m=+6.163643271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h8fzq" (UniqueName: "kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq") pod "network-check-target-6mlp5" (UID: "af87ecf6-e9d9-4c65-ace2-bc1709237610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:17.374490 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.374371 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:25:15 +0000 UTC" deadline="2028-03-18 02:38:19.208531299 +0000 UTC" Sep 17 15:30:17.374490 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.374437 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13139h8m1.834098341s" Sep 17 15:30:17.459317 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.458322 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:17.459317 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:17.458529 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:17.459317 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.458683 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:17.459317 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:17.458783 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:17.459317 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.458849 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:17.459317 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:17.458921 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:17.498675 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.498638 2581 generic.go:356] "Generic (PLEG): container finished" podID="6ee901a15dc913f04af8abf50b96b53a" containerID="1dc3f7c5f4b6916d52b9a6f245cc46f1a4c176f6bf9ab0b99904cae0d00a43f1" exitCode=0 Sep 17 15:30:17.498845 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.498733 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal" event={"ID":"6ee901a15dc913f04af8abf50b96b53a","Type":"ContainerDied","Data":"1dc3f7c5f4b6916d52b9a6f245cc46f1a4c176f6bf9ab0b99904cae0d00a43f1"} Sep 17 15:30:17.510831 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.510800 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-15.ec2.internal" event={"ID":"e96dc0286b5d333a4741dcc145860b91","Type":"ContainerStarted","Data":"966dabd5679df6e19e6de4dbcf5cdd681b23f15ffc7960b6b01d4089b5950c3f"} Sep 17 15:30:17.522932 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.522902 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wd878" event={"ID":"30c3098f-6b70-43ef-96fe-67a7c9f4ff91","Type":"ContainerStarted","Data":"e1f592f6a83805c5c9476e19c3855616fdb1e7ce41a2c154c94a5a261aa465bc"} Sep 17 15:30:17.550711 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.550677 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" event={"ID":"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae","Type":"ContainerStarted","Data":"4a5d131e8fb3a4d995f8aae7fd0366664a31fba50b4719adaa97591ae9b873d6"} Sep 17 15:30:17.559127 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.559098 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d7hx7" event={"ID":"0402d124-60fa-4a15-86cc-cac6e6904bb3","Type":"ContainerStarted","Data":"d5262136ad2dc389b674349dde83a62161cad7677709fd133b8dfd2f52db0c54"} Sep 17 15:30:17.594445 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.594187 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5r65k" event={"ID":"680c1df0-298c-464a-a1cf-4739d24ba0fa","Type":"ContainerStarted","Data":"148672b45e229931fea73f80afe822973db1e68f56596d8ad0fd0b11209a314e"} Sep 17 15:30:17.602352 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.602320 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lv4s5" event={"ID":"18b5b4bc-a399-468b-9957-2d51f1f24c6c","Type":"ContainerStarted","Data":"23bd5c7447e09262a41a358b1a514ef7ef20dfc0f6aed4ef7e618d24654f6289"} Sep 17 15:30:17.606736 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.606709 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" event={"ID":"573f3972-eb62-40d9-aff1-c1baf1d15362","Type":"ContainerStarted","Data":"9c0202765b7b6fb7a78db486d7d5bac61ac0f3d827ff5cafc43509861822c1d5"} Sep 17 15:30:17.621224 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.621195 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" event={"ID":"3e716979-d391-4561-b731-64cd1b6efed3","Type":"ContainerStarted","Data":"23acb8ace595529e3e020089a4ecff77d596881d6d143f944ae38305a33bac15"} Sep 17 15:30:17.633711 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.633605 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-44mlh" event={"ID":"9f68347c-394c-4828-b962-ffe2684471cb","Type":"ContainerStarted","Data":"317a268b736d14cf228657078904211bb9a30fe106153ea4856303b0e06b4d31"} Sep 17 15:30:17.660479 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:17.660444 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxcg" event={"ID":"899f7ef8-d770-4340-bd86-7552cd1ac461","Type":"ContainerStarted","Data":"970ba94ae4f50bb8b9d5a2564c2c3ada5094f97b570292acda28c195729936e5"} Sep 17 15:30:18.720820 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:18.720400 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal" event={"ID":"6ee901a15dc913f04af8abf50b96b53a","Type":"ContainerStarted","Data":"422ab8ddbca92d283eea103c6bb7e1e02ec8b4b54c14f86d7000116cbf763eac"} Sep 17 15:30:18.738451 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:18.738390 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-15.ec2.internal" podStartSLOduration=4.738366277 podStartE2EDuration="4.738366277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:17.540876419 +0000 UTC m=+4.537113169" watchObservedRunningTime="2026-09-17 15:30:18.738366277 +0000 UTC m=+5.734603028" Sep 17 15:30:18.981746 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:18.981651 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:18.981905 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:18.981865 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:18.981995 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:18.981931 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret podName:721e805e-1d0a-473c-be1a-871d62cfebbe nodeName:}" failed. No retries permitted until 2026-09-17 15:30:22.981907084 +0000 UTC m=+9.978143822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret") pod "global-pull-secret-syncer-5rf6m" (UID: "721e805e-1d0a-473c-be1a-871d62cfebbe") : object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:19.082323 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:19.082288 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs\") pod \"network-metrics-daemon-cgcpl\" (UID: \"fa257b5d-59d6-4286-a99f-b2bee16de288\") " pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:19.082594 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:19.082573 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:19.082689 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:19.082648 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs podName:fa257b5d-59d6-4286-a99f-b2bee16de288 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:23.082628584 +0000 UTC m=+10.078865312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs") pod "network-metrics-daemon-cgcpl" (UID: "fa257b5d-59d6-4286-a99f-b2bee16de288") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:19.183644 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:19.183603 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fzq\" (UniqueName: \"kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq\") pod \"network-check-target-6mlp5\" (UID: \"af87ecf6-e9d9-4c65-ace2-bc1709237610\") " pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:19.183916 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:19.183882 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 15:30:19.184045 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:19.183911 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 15:30:19.184117 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:19.184051 2581 projected.go:196] Error preparing data for projected volume kube-api-access-h8fzq for pod openshift-network-diagnostics/network-check-target-6mlp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:19.184187 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:19.184118 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq podName:af87ecf6-e9d9-4c65-ace2-bc1709237610 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:23.184098043 +0000 UTC m=+10.180334772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h8fzq" (UniqueName: "kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq") pod "network-check-target-6mlp5" (UID: "af87ecf6-e9d9-4c65-ace2-bc1709237610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:19.462191 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:19.462103 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:19.462351 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:19.462286 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:19.463015 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:19.462949 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:19.463148 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:19.463124 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:19.463264 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:19.463250 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:19.463396 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:19.463376 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:21.456944 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:21.456428 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:21.456944 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:21.456549 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:21.456944 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:21.456942 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:21.457503 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:21.457053 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:21.457503 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:21.457065 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:21.457503 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:21.457168 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:23.015827 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:23.015718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:23.016311 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:23.015836 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:23.016311 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:23.015902 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret podName:721e805e-1d0a-473c-be1a-871d62cfebbe nodeName:}" failed. No retries permitted until 2026-09-17 15:30:31.015884566 +0000 UTC m=+18.012121303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret") pod "global-pull-secret-syncer-5rf6m" (UID: "721e805e-1d0a-473c-be1a-871d62cfebbe") : object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:23.116493 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:23.116453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs\") pod \"network-metrics-daemon-cgcpl\" (UID: \"fa257b5d-59d6-4286-a99f-b2bee16de288\") " pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:23.116673 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:23.116626 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:23.116741 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:23.116703 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs podName:fa257b5d-59d6-4286-a99f-b2bee16de288 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:31.116682204 +0000 UTC m=+18.112918934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs") pod "network-metrics-daemon-cgcpl" (UID: "fa257b5d-59d6-4286-a99f-b2bee16de288") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:23.217508 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:23.217457 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fzq\" (UniqueName: \"kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq\") pod \"network-check-target-6mlp5\" (UID: \"af87ecf6-e9d9-4c65-ace2-bc1709237610\") " pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:23.217673 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:23.217632 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 15:30:23.217673 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:23.217652 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 15:30:23.217673 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:23.217664 2581 projected.go:196] Error preparing data for projected volume kube-api-access-h8fzq for pod openshift-network-diagnostics/network-check-target-6mlp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:23.217847 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:23.217717 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq podName:af87ecf6-e9d9-4c65-ace2-bc1709237610 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:31.217704272 +0000 UTC m=+18.213941009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h8fzq" (UniqueName: "kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq") pod "network-check-target-6mlp5" (UID: "af87ecf6-e9d9-4c65-ace2-bc1709237610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:23.457107 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:23.456970 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:23.457107 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:23.457096 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:23.457409 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:23.457380 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:23.457524 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:23.457439 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:23.457581 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:23.457530 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:23.457639 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:23.457599 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:25.456935 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:25.456885 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:25.457398 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:25.457035 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:25.457398 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:25.457037 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:25.457398 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:25.457091 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:25.457398 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:25.457159 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:25.457398 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:25.457231 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:27.456243 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:27.456162 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:27.456243 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:27.456192 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:27.456736 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:27.456162 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:27.456736 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:27.456276 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:27.456736 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:27.456349 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:27.456736 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:27.456461 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:29.456782 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:29.456744 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:29.457280 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:29.456788 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:29.457280 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:29.456749 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:29.457280 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:29.456874 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:29.457280 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:29.456964 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:29.457280 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:29.457060 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:31.071096 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:31.071043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:31.071508 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:31.071193 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:31.071508 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:31.071269 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret podName:721e805e-1d0a-473c-be1a-871d62cfebbe nodeName:}" failed. No retries permitted until 2026-09-17 15:30:47.071248203 +0000 UTC m=+34.067484947 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret") pod "global-pull-secret-syncer-5rf6m" (UID: "721e805e-1d0a-473c-be1a-871d62cfebbe") : object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:31.171606 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:31.171567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs\") pod \"network-metrics-daemon-cgcpl\" (UID: \"fa257b5d-59d6-4286-a99f-b2bee16de288\") " pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:31.171798 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:31.171739 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:31.171859 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:31.171806 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs podName:fa257b5d-59d6-4286-a99f-b2bee16de288 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:47.171790947 +0000 UTC m=+34.168027671 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs") pod "network-metrics-daemon-cgcpl" (UID: "fa257b5d-59d6-4286-a99f-b2bee16de288") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:31.272943 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:31.272894 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fzq\" (UniqueName: \"kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq\") pod \"network-check-target-6mlp5\" (UID: \"af87ecf6-e9d9-4c65-ace2-bc1709237610\") " pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:31.273083 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:31.273065 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 15:30:31.273137 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:31.273087 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 15:30:31.273137 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:31.273097 2581 projected.go:196] Error preparing data for projected volume kube-api-access-h8fzq for pod openshift-network-diagnostics/network-check-target-6mlp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:31.273211 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:31.273155 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq podName:af87ecf6-e9d9-4c65-ace2-bc1709237610 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:47.273140391 +0000 UTC m=+34.269377117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h8fzq" (UniqueName: "kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq") pod "network-check-target-6mlp5" (UID: "af87ecf6-e9d9-4c65-ace2-bc1709237610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:31.456620 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:31.456539 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:31.456620 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:31.456554 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:31.456811 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:31.456673 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:31.456811 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:31.456718 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:31.456811 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:31.456747 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:31.456928 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:31.456823 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:33.456633 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:33.456603 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:33.457102 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:33.456753 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:33.457102 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:33.456772 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:33.457102 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:33.456837 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:33.457102 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:33.456848 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:33.457102 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:33.456945 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:34.754241 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.754067 2581 generic.go:356] "Generic (PLEG): container finished" podID="899f7ef8-d770-4340-bd86-7552cd1ac461" containerID="d960756e3bd3d05e09f0d6ed75b7c95fbf9f21638a317d6e3e201cbcfcf2855c" exitCode=0 Sep 17 15:30:34.754986 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.754112 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxcg" event={"ID":"899f7ef8-d770-4340-bd86-7552cd1ac461","Type":"ContainerDied","Data":"d960756e3bd3d05e09f0d6ed75b7c95fbf9f21638a317d6e3e201cbcfcf2855c"} Sep 17 15:30:34.755853 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.755828 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wd878" event={"ID":"30c3098f-6b70-43ef-96fe-67a7c9f4ff91","Type":"ContainerStarted","Data":"73a7fe998f392eefd0cfb0ba5e03077d7d75528ad43d4687009995ac31d90408"} Sep 17 15:30:34.757428 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.757405 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" event={"ID":"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae","Type":"ContainerStarted","Data":"099a10ebd6f312f744947578a837d792db315f7fdb8850af96b45939ec467568"} Sep 17 15:30:34.758738 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.758717 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d7hx7" event={"ID":"0402d124-60fa-4a15-86cc-cac6e6904bb3","Type":"ContainerStarted","Data":"0787fefbb19c5cbe33e0e24a3029ce14b58282ec57b2a7afee77c9185a14a1e3"} Sep 17 15:30:34.760181 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.760157 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5r65k" event={"ID":"680c1df0-298c-464a-a1cf-4739d24ba0fa","Type":"ContainerStarted","Data":"1d25ca32eab1ab3b91395a97cf8be02be5586a53c9b1aa100bc9bb68d1f0e891"} Sep 17 15:30:34.761594 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.761573 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lv4s5" event={"ID":"18b5b4bc-a399-468b-9957-2d51f1f24c6c","Type":"ContainerStarted","Data":"171be6d715b5c1ba9b5debd3e8e08698ddc9ce81cb43a06b5eb7ac177cc03741"} Sep 17 15:30:34.763601 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.763581 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" event={"ID":"573f3972-eb62-40d9-aff1-c1baf1d15362","Type":"ContainerStarted","Data":"380956df302186ee09236074564f917f6c5cb75ccb805c3d82f261bf032bc135"} Sep 17 15:30:34.763681 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.763642 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" event={"ID":"573f3972-eb62-40d9-aff1-c1baf1d15362","Type":"ContainerStarted","Data":"5061df65fef82e99341d0daed82684f7132a7f1822869d9faa931acde4239dfd"} Sep 17 15:30:34.763751 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.763718 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" event={"ID":"573f3972-eb62-40d9-aff1-c1baf1d15362","Type":"ContainerStarted","Data":"0f0d5cf1d0a7df7b3ceda89d3ad4cf0a9de45efca624686f2da0b4d96e6e869a"} Sep 17 15:30:34.764886 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.764869 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" event={"ID":"3e716979-d391-4561-b731-64cd1b6efed3","Type":"ContainerStarted","Data":"8b9a5bb12e33ff93d3a6e566393f670cfe587a95fa9174bf13bbaeb5075224a1"} Sep 17 15:30:34.774487 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.774375 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-15.ec2.internal" podStartSLOduration=20.77436343 podStartE2EDuration="20.77436343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:18.739555032 +0000 UTC m=+5.735791796" watchObservedRunningTime="2026-09-17 15:30:34.77436343 +0000 UTC m=+21.770600176" Sep 17 15:30:34.791486 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.791438 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-x2wwn" podStartSLOduration=3.000876954 podStartE2EDuration="20.79142125s" totalImagesPullingTime="17.790544296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:34.791416929 +0000 UTC m=+21.787653675" watchObservedRunningTime="2026-09-17 15:30:34.79142125 +0000 UTC m=+21.787658056" Sep 17 15:30:34.807112 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.807077 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lv4s5" podStartSLOduration=10.258908232 podStartE2EDuration="20.807062595s" totalImagesPullingTime="10.548154363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:34.806860516 +0000 UTC m=+21.803097263" watchObservedRunningTime="2026-09-17 15:30:34.807062595 +0000 UTC m=+21.803299347" Sep 17 15:30:34.823943 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.823887 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d7hx7" podStartSLOduration=3.035205416 podStartE2EDuration="20.823872655s" totalImagesPullingTime="17.788667239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:34.823873538 +0000 UTC m=+21.820110280" watchObservedRunningTime="2026-09-17 15:30:34.823872655 +0000 UTC m=+21.820109400" Sep 17 15:30:34.845409 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.845371 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5r65k" podStartSLOduration=3.001843626 podStartE2EDuration="20.845357283s" totalImagesPullingTime="17.843513657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:34.845036176 +0000 UTC m=+21.841272921" watchObservedRunningTime="2026-09-17 15:30:34.845357283 +0000 UTC m=+21.841594030" Sep 17 15:30:34.860696 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.860661 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wd878" podStartSLOduration=3.074628098 podStartE2EDuration="20.860648371s" totalImagesPullingTime="17.786020273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:34.86052562 +0000 UTC m=+21.856762367" watchObservedRunningTime="2026-09-17 15:30:34.860648371 +0000 UTC m=+21.856885116" Sep 17 15:30:34.885372 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:34.885347 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:35.085843 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:35.085749 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:35.280173 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:35.280144 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:35.456355 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:35.456243 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:35.456501 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:35.456250 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:35.456501 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:35.456379 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:35.456501 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:35.456251 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:35.456654 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:35.456550 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:35.456654 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:35.456463 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:35.771201 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:35.771114 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" event={"ID":"573f3972-eb62-40d9-aff1-c1baf1d15362","Type":"ContainerStarted","Data":"e616036cd301c99157c7e02e5175b5b03f00cfc9ee3263068d41dc742b004d05"} Sep 17 15:30:35.771662 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:35.771216 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" event={"ID":"573f3972-eb62-40d9-aff1-c1baf1d15362","Type":"ContainerStarted","Data":"320210ad0788e2d1a2e9aa084e53a3b24cedbe91f349479dc9e3229bbc1fa035"} Sep 17 15:30:35.771662 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:35.771260 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" event={"ID":"573f3972-eb62-40d9-aff1-c1baf1d15362","Type":"ContainerStarted","Data":"93b516c1c1492a057ea3373cbbd82fca83535207482f39c021e651b8b0758cf3"} Sep 17 15:30:35.773360 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:35.773298 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-44mlh" event={"ID":"9f68347c-394c-4828-b962-ffe2684471cb","Type":"ContainerStarted","Data":"0a7fdabdfd101dbac843e2b7d3a6c3757aa96e994931809ebbf07bd692e74914"} Sep 17 15:30:35.877769 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:35.877741 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:36.108693 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:36.108555 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:36.211467 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:36.211437 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 15:30:36.384821 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:36.384678 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:30:36.211458438Z","UUID":"62dff0b7-010f-40f2-a67e-6f62cc007dab","Handler":null,"Name":"","Endpoint":""} Sep 17 15:30:36.387033 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:36.387005 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 15:30:36.387139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:36.387048 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 15:30:36.483665 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:36.483633 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:36.703494 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:36.703419 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:36.777727 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:36.777695 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" event={"ID":"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae","Type":"ContainerStarted","Data":"1a834c11a14514d90b1234e8ecdcea591853c34dc02a9edc9b5fa8328fd1f424"} Sep 17 15:30:36.957138 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:36.957046 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:37.060691 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.060657 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:37.159548 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.159515 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:37.187655 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.187626 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lv4s5" Sep 17 15:30:37.188542 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.188514 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lv4s5" Sep 17 15:30:37.204551 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.204504 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-44mlh" podStartSLOduration=5.407658458 podStartE2EDuration="23.204484273s" totalImagesPullingTime="17.796825815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:35.789673254 +0000 UTC m=+22.785910001" watchObservedRunningTime="2026-09-17 15:30:37.204484273 +0000 UTC m=+24.200721021" Sep 17 15:30:37.280807 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.280727 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:37.459886 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.459865 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:37.460011 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.459866 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:37.460011 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.459875 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:37.460091 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:37.460028 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:37.460133 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:37.460103 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:37.460133 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:37.459990 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:37.782317 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.782289 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" event={"ID":"07ffcf0a-8dd5-4289-8c98-b1bbb48270ae","Type":"ContainerStarted","Data":"2b78c5060b95056a5f32afc55804ce370caa8d1b9bccd224e401766c9785abf5"} Sep 17 15:30:37.786129 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.786105 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" event={"ID":"573f3972-eb62-40d9-aff1-c1baf1d15362","Type":"ContainerStarted","Data":"b0fac4587576b59dca61f3a800a5e51351bd2646824f59189d5f028a6a85c226"} Sep 17 15:30:37.786257 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.786171 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lv4s5" Sep 17 15:30:37.786643 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.786627 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lv4s5" Sep 17 15:30:37.799784 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:37.799752 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6mhqp" podStartSLOduration=3.240107109 podStartE2EDuration="23.799739192s" totalImagesPullingTime="20.559632083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:37.798950486 +0000 UTC m=+24.795187232" watchObservedRunningTime="2026-09-17 15:30:37.799739192 +0000 UTC m=+24.795975937" Sep 17 15:30:38.079630 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:38.079564 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:38.377837 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:38.377755 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:38.483556 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:38.483525 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:38.882560 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:38.882348 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:39.079104 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:39.079069 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:39.456620 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:39.456539 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:39.456620 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:39.456581 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:39.456844 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:39.456548 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:39.456844 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:39.456681 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:39.456844 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:39.456770 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:39.457013 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:39.456854 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:39.709954 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:39.709935 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:39.795427 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:39.795336 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" event={"ID":"573f3972-eb62-40d9-aff1-c1baf1d15362","Type":"ContainerStarted","Data":"71927cde273786b6fed3ca2069d12b87d2bd75a63679ff33b02527c3a1c9682d"} Sep 17 15:30:40.282263 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:40.282198 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:40.474583 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:40.474554 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:40.684705 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:40.684479 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:40.798947 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:40.798901 2581 generic.go:356] "Generic (PLEG): container finished" podID="899f7ef8-d770-4340-bd86-7552cd1ac461" containerID="f822f636c978cbc23f8c925502389b5dca09bbd6f856d2ee335177519f8f7a25" exitCode=0 Sep 17 15:30:40.799104 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:40.798982 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxcg" event={"ID":"899f7ef8-d770-4340-bd86-7552cd1ac461","Type":"ContainerDied","Data":"f822f636c978cbc23f8c925502389b5dca09bbd6f856d2ee335177519f8f7a25"} Sep 17 15:30:40.799250 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:40.799224 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:40.799375 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:40.799303 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:40.799375 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:40.799353 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:40.814655 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:40.814634 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:40.814763 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:40.814727 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:30:40.873738 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:40.873685 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" podStartSLOduration=9.020894893 podStartE2EDuration="26.873667955s" totalImagesPullingTime="17.852773062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:40.873035791 +0000 UTC m=+27.869272534" watchObservedRunningTime="2026-09-17 15:30:40.873667955 +0000 UTC m=+27.869904716" Sep 17 15:30:41.081988 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:41.081869 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:41.389151 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:41.389079 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:41.456310 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:41.456273 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:41.456485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:41.456268 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:41.456485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:41.456419 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:41.456485 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:41.456414 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:41.456605 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:41.456525 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:41.456652 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:41.456610 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:41.679708 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:41.679637 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:42.132175 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:42.132146 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:42.296969 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:42.296876 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:42.694989 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:42.694531 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:42.795628 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:42.795600 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:42.804222 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:42.804193 2581 generic.go:356] "Generic (PLEG): container finished" podID="899f7ef8-d770-4340-bd86-7552cd1ac461" containerID="260427ecef30fdf4527643325929f1ac8cbada7cf7b9b65e131a4dde546f2978" exitCode=0 Sep 17 15:30:42.804362 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:42.804269 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxcg" event={"ID":"899f7ef8-d770-4340-bd86-7552cd1ac461","Type":"ContainerDied","Data":"260427ecef30fdf4527643325929f1ac8cbada7cf7b9b65e131a4dde546f2978"} Sep 17 15:30:42.894749 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:42.894722 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:43.456754 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:43.456732 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:43.456843 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:43.456809 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:43.456958 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:43.456936 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:43.457015 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:43.456973 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:43.457113 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:43.457065 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:43.457146 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:43.457126 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:43.809387 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:43.809312 2581 generic.go:356] "Generic (PLEG): container finished" podID="899f7ef8-d770-4340-bd86-7552cd1ac461" containerID="6b24a4092ec4b344ab72e86ed264e97bcf8b0de486d9f62d695e3898fee2705d" exitCode=0 Sep 17 15:30:43.809387 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:43.809359 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxcg" event={"ID":"899f7ef8-d770-4340-bd86-7552cd1ac461","Type":"ContainerDied","Data":"6b24a4092ec4b344ab72e86ed264e97bcf8b0de486d9f62d695e3898fee2705d"} Sep 17 15:30:44.669584 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:44.669549 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:44.770406 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:44.770366 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:44.968566 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:44.968486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:45.346589 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:45.346515 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:45.447241 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:45.447209 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:45.456279 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:45.456255 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:45.456406 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:45.456260 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:45.456406 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:45.456349 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:45.456406 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:45.456362 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:45.456568 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:45.456444 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:45.456568 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:45.456512 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:45.760767 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:45.760691 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:46.362941 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:46.362891 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:46.418268 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:46.418239 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:46.523321 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:46.523277 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:46.624264 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:46.624189 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:47.097791 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:47.097729 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:47.097945 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:47.097881 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:47.098005 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:47.097956 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret podName:721e805e-1d0a-473c-be1a-871d62cfebbe nodeName:}" failed. No retries permitted until 2026-09-17 15:31:19.097938399 +0000 UTC m=+66.094175132 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret") pod "global-pull-secret-syncer-5rf6m" (UID: "721e805e-1d0a-473c-be1a-871d62cfebbe") : object "kube-system"/"original-pull-secret" not registered Sep 17 15:30:47.198168 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:47.198135 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs\") pod \"network-metrics-daemon-cgcpl\" (UID: \"fa257b5d-59d6-4286-a99f-b2bee16de288\") " pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:47.198306 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:47.198243 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:47.198306 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:47.198288 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs podName:fa257b5d-59d6-4286-a99f-b2bee16de288 nodeName:}" failed. No retries permitted until 2026-09-17 15:31:19.198275097 +0000 UTC m=+66.194511825 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs") pod "network-metrics-daemon-cgcpl" (UID: "fa257b5d-59d6-4286-a99f-b2bee16de288") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 15:30:47.299313 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:47.299277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fzq\" (UniqueName: \"kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq\") pod \"network-check-target-6mlp5\" (UID: \"af87ecf6-e9d9-4c65-ace2-bc1709237610\") " pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:47.299505 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:47.299411 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 15:30:47.299505 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:47.299429 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 15:30:47.299505 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:47.299437 2581 projected.go:196] Error preparing data for projected volume kube-api-access-h8fzq for pod openshift-network-diagnostics/network-check-target-6mlp5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:47.299505 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:47.299486 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq podName:af87ecf6-e9d9-4c65-ace2-bc1709237610 nodeName:}" failed. No retries permitted until 2026-09-17 15:31:19.299470127 +0000 UTC m=+66.295706863 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h8fzq" (UniqueName: "kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq") pod "network-check-target-6mlp5" (UID: "af87ecf6-e9d9-4c65-ace2-bc1709237610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 15:30:47.456073 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:47.455982 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:47.456073 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:47.456047 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:47.456073 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:47.455996 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:47.456616 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:47.456169 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:47.456616 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:47.456251 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:47.456616 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:47.456608 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:47.875810 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:47.875733 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:47.933715 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:47.933680 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:48.778648 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:48.778613 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:48.883815 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:48.883783 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:49.456185 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:49.456122 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:49.456185 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:49.456144 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:49.456343 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:49.456124 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:49.456343 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:49.456224 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:49.456343 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:49.456283 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:49.456453 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:49.456359 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:49.826151 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:49.826121 2581 generic.go:356] "Generic (PLEG): container finished" podID="899f7ef8-d770-4340-bd86-7552cd1ac461" containerID="40bfc924bdbf3f7fc3026acb5352e0112d991ff383cb64dafbc3df4c5e113e51" exitCode=0 Sep 17 15:30:49.826520 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:49.826166 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxcg" event={"ID":"899f7ef8-d770-4340-bd86-7552cd1ac461","Type":"ContainerDied","Data":"40bfc924bdbf3f7fc3026acb5352e0112d991ff383cb64dafbc3df4c5e113e51"} Sep 17 15:30:50.830755 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:50.830726 2581 generic.go:356] "Generic (PLEG): container finished" podID="899f7ef8-d770-4340-bd86-7552cd1ac461" containerID="9141dc8d02570b58d834e447e67dbd820148d142ce4ab61b2b203bfaf067a377" exitCode=0 Sep 17 15:30:50.831114 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:50.830761 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxcg" event={"ID":"899f7ef8-d770-4340-bd86-7552cd1ac461","Type":"ContainerDied","Data":"9141dc8d02570b58d834e447e67dbd820148d142ce4ab61b2b203bfaf067a377"} Sep 17 15:30:51.456535 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:51.456506 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:51.456535 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:51.456517 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:51.456774 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:51.456517 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:51.456774 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:51.456605 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:51.456774 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:51.456682 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:51.456774 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:51.456739 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:51.841227 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:51.841194 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lsxcg" event={"ID":"899f7ef8-d770-4340-bd86-7552cd1ac461","Type":"ContainerStarted","Data":"5f55695ebf0c92b2d80c17447bae1b8e898767eed2c9d895d2a3d3655b236fef"} Sep 17 15:30:52.001082 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:52.001014 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:52.195505 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:52.195436 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:52.301582 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:52.301556 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:52.394865 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:52.394836 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:52.822989 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:52.822609 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lsxcg" podStartSLOduration=16.066322937 podStartE2EDuration="38.822586468s" totalImagesPullingTime="22.756263531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:51.884817332 +0000 UTC m=+38.881054078" watchObservedRunningTime="2026-09-17 15:30:52.822586468 +0000 UTC m=+39.818823225" Sep 17 15:30:52.823349 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:52.823322 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5rf6m"] Sep 17 15:30:52.823455 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:52.823443 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:52.823551 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:52.823534 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:52.827472 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:52.827409 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6mlp5"] Sep 17 15:30:52.827582 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:52.827534 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:52.827649 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:52.827626 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:52.828024 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:52.827997 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cgcpl"] Sep 17 15:30:52.828134 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:52.828105 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:52.828230 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:52.828210 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:54.456123 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:54.456091 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:54.456681 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:54.456129 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:54.456681 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:54.456202 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:54.456681 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:54.456250 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:54.456681 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:54.456322 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:54.456681 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:54.456374 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:56.455736 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:56.455700 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:56.455736 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:56.455724 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:56.456197 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:56.455812 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:56.456197 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:56.455820 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5rf6m" podUID="721e805e-1d0a-473c-be1a-871d62cfebbe" Sep 17 15:30:56.456197 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:56.455864 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6mlp5" podUID="af87ecf6-e9d9-4c65-ace2-bc1709237610" Sep 17 15:30:56.456197 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:30:56.455984 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cgcpl" podUID="fa257b5d-59d6-4286-a99f-b2bee16de288" Sep 17 15:30:57.313975 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.313948 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-15.ec2.internal" event="NodeReady" Sep 17 15:30:57.314151 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.314049 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 15:30:57.352686 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.352654 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65d59949f4-lb8rv"] Sep 17 15:30:57.376864 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.376840 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h5587"] Sep 17 15:30:57.377032 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.376992 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.379025 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.379000 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 15:30:57.379160 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.379140 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 15:30:57.379227 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.379144 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lzxqs\"" Sep 17 15:30:57.379419 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.379403 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 15:30:57.384249 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.384230 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 15:30:57.384596 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.384578 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65d59949f4-lb8rv"] Sep 17 15:30:57.384680 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.384613 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ds2t5"] Sep 17 15:30:57.384721 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.384680 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.387004 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.386984 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 15:30:57.387117 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.387038 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 15:30:57.387292 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.387148 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l7hr4\"" Sep 17 15:30:57.395971 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.394823 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h5587"] Sep 17 15:30:57.395971 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.395641 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ds2t5"] Sep 17 15:30:57.398043 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.398023 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.401201 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.401185 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 15:30:57.401552 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.401534 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 15:30:57.401879 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.401859 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hk45w\"" Sep 17 15:30:57.401978 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.401864 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 15:30:57.402026 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.401987 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 15:30:57.471283 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.471258 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jcm4q"] Sep 17 15:30:57.490242 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.490222 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jcm4q"] Sep 17 15:30:57.490361 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.490298 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jcm4q" Sep 17 15:30:57.493615 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.493595 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 15:30:57.493707 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.493656 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 15:30:57.493707 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.493664 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 15:30:57.494018 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.494000 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bcwx7\"" Sep 17 15:30:57.515017 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.514980 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:57.572030 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.571980 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3816cff-380c-4f26-bc82-a648050bea57-data-volume\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.572030 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572009 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/06863d44-90d7-4d15-a6df-5597dcb22945-image-registry-private-configuration\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.572157 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572039 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3816cff-380c-4f26-bc82-a648050bea57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.572157 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572058 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06863d44-90d7-4d15-a6df-5597dcb22945-registry-tls\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.572157 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06863d44-90d7-4d15-a6df-5597dcb22945-trusted-ca\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.572157 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572091 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a766297c-8356-4831-815f-52186c63cd92-metrics-tls\") pod \"dns-default-h5587\" (UID: \"a766297c-8356-4831-815f-52186c63cd92\") " pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.572157 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572112 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/06863d44-90d7-4d15-a6df-5597dcb22945-registry-certificates\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.572157 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3816cff-380c-4f26-bc82-a648050bea57-crio-socket\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.572157 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572141 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/06863d44-90d7-4d15-a6df-5597dcb22945-installation-pull-secrets\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.572486 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572196 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3816cff-380c-4f26-bc82-a648050bea57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.572486 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572246 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24wk8\" (UniqueName: \"kubernetes.io/projected/b3816cff-380c-4f26-bc82-a648050bea57-kube-api-access-24wk8\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.572486 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a766297c-8356-4831-815f-52186c63cd92-config-volume\") pod \"dns-default-h5587\" (UID: \"a766297c-8356-4831-815f-52186c63cd92\") " pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.572486 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572321 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th6tf\" (UniqueName: \"kubernetes.io/projected/a766297c-8356-4831-815f-52186c63cd92-kube-api-access-th6tf\") pod \"dns-default-h5587\" (UID: \"a766297c-8356-4831-815f-52186c63cd92\") " pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.572486 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572350 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjkdd\" (UniqueName: \"kubernetes.io/projected/06863d44-90d7-4d15-a6df-5597dcb22945-kube-api-access-bjkdd\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.572486 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572375 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a766297c-8356-4831-815f-52186c63cd92-tmp-dir\") pod \"dns-default-h5587\" (UID: \"a766297c-8356-4831-815f-52186c63cd92\") " pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.572486 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572420 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06863d44-90d7-4d15-a6df-5597dcb22945-bound-sa-token\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.572486 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.572458 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/06863d44-90d7-4d15-a6df-5597dcb22945-ca-trust-extracted\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.581746 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.581724 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:57.673334 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/06863d44-90d7-4d15-a6df-5597dcb22945-image-registry-private-configuration\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.673498 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673351 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3816cff-380c-4f26-bc82-a648050bea57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.673498 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673371 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06863d44-90d7-4d15-a6df-5597dcb22945-registry-tls\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.673498 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673394 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06863d44-90d7-4d15-a6df-5597dcb22945-trusted-ca\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.673498 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673421 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a766297c-8356-4831-815f-52186c63cd92-metrics-tls\") pod \"dns-default-h5587\" (UID: \"a766297c-8356-4831-815f-52186c63cd92\") " pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.673498 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673457 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/06863d44-90d7-4d15-a6df-5597dcb22945-registry-certificates\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.673498 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673482 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3816cff-380c-4f26-bc82-a648050bea57-crio-socket\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.673772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673503 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/06863d44-90d7-4d15-a6df-5597dcb22945-installation-pull-secrets\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.673772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673530 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78acfb4b-12cd-4266-9f36-3c13f8a18263-cert\") pod \"ingress-canary-jcm4q\" (UID: \"78acfb4b-12cd-4266-9f36-3c13f8a18263\") " pod="openshift-ingress-canary/ingress-canary-jcm4q" Sep 17 15:30:57.673772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3816cff-380c-4f26-bc82-a648050bea57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.673772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673589 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24wk8\" (UniqueName: \"kubernetes.io/projected/b3816cff-380c-4f26-bc82-a648050bea57-kube-api-access-24wk8\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.673772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673617 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a766297c-8356-4831-815f-52186c63cd92-config-volume\") pod \"dns-default-h5587\" (UID: \"a766297c-8356-4831-815f-52186c63cd92\") " pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.673772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th6tf\" (UniqueName: \"kubernetes.io/projected/a766297c-8356-4831-815f-52186c63cd92-kube-api-access-th6tf\") pod \"dns-default-h5587\" (UID: \"a766297c-8356-4831-815f-52186c63cd92\") " pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.673772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673666 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grfff\" (UniqueName: \"kubernetes.io/projected/78acfb4b-12cd-4266-9f36-3c13f8a18263-kube-api-access-grfff\") pod \"ingress-canary-jcm4q\" (UID: \"78acfb4b-12cd-4266-9f36-3c13f8a18263\") " pod="openshift-ingress-canary/ingress-canary-jcm4q" Sep 17 15:30:57.673772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjkdd\" (UniqueName: \"kubernetes.io/projected/06863d44-90d7-4d15-a6df-5597dcb22945-kube-api-access-bjkdd\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.673772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673712 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a766297c-8356-4831-815f-52186c63cd92-tmp-dir\") pod \"dns-default-h5587\" (UID: \"a766297c-8356-4831-815f-52186c63cd92\") " pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.673772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06863d44-90d7-4d15-a6df-5597dcb22945-bound-sa-token\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.673772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673774 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/06863d44-90d7-4d15-a6df-5597dcb22945-ca-trust-extracted\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.674346 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.673806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3816cff-380c-4f26-bc82-a648050bea57-data-volume\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.674346 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.674137 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3816cff-380c-4f26-bc82-a648050bea57-data-volume\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.674346 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.674303 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3816cff-380c-4f26-bc82-a648050bea57-crio-socket\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.674471 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.674412 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06863d44-90d7-4d15-a6df-5597dcb22945-trusted-ca\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.674795 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.674774 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/06863d44-90d7-4d15-a6df-5597dcb22945-registry-certificates\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.674927 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.674889 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a766297c-8356-4831-815f-52186c63cd92-config-volume\") pod \"dns-default-h5587\" (UID: \"a766297c-8356-4831-815f-52186c63cd92\") " pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.675027 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.675002 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3816cff-380c-4f26-bc82-a648050bea57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.675076 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.675041 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a766297c-8356-4831-815f-52186c63cd92-tmp-dir\") pod \"dns-default-h5587\" (UID: \"a766297c-8356-4831-815f-52186c63cd92\") " pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.675435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.675413 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/06863d44-90d7-4d15-a6df-5597dcb22945-ca-trust-extracted\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.677654 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.677629 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3816cff-380c-4f26-bc82-a648050bea57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.677654 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.677644 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/06863d44-90d7-4d15-a6df-5597dcb22945-image-registry-private-configuration\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.677781 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.677664 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a766297c-8356-4831-815f-52186c63cd92-metrics-tls\") pod \"dns-default-h5587\" (UID: \"a766297c-8356-4831-815f-52186c63cd92\") " pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.677781 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.677757 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06863d44-90d7-4d15-a6df-5597dcb22945-registry-tls\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.677781 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.677771 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/06863d44-90d7-4d15-a6df-5597dcb22945-installation-pull-secrets\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.682672 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.682642 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06863d44-90d7-4d15-a6df-5597dcb22945-bound-sa-token\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.682948 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.682932 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjkdd\" (UniqueName: \"kubernetes.io/projected/06863d44-90d7-4d15-a6df-5597dcb22945-kube-api-access-bjkdd\") pod \"image-registry-65d59949f4-lb8rv\" (UID: \"06863d44-90d7-4d15-a6df-5597dcb22945\") " pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.683034 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.682950 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24wk8\" (UniqueName: \"kubernetes.io/projected/b3816cff-380c-4f26-bc82-a648050bea57-kube-api-access-24wk8\") pod \"insights-runtime-extractor-ds2t5\" (UID: \"b3816cff-380c-4f26-bc82-a648050bea57\") " pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.683228 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.683213 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th6tf\" (UniqueName: \"kubernetes.io/projected/a766297c-8356-4831-815f-52186c63cd92-kube-api-access-th6tf\") pod \"dns-default-h5587\" (UID: \"a766297c-8356-4831-815f-52186c63cd92\") " pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.689141 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.689125 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:57.699817 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.699800 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h5587" Sep 17 15:30:57.706970 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.706949 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ds2t5" Sep 17 15:30:57.775214 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.775175 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78acfb4b-12cd-4266-9f36-3c13f8a18263-cert\") pod \"ingress-canary-jcm4q\" (UID: \"78acfb4b-12cd-4266-9f36-3c13f8a18263\") " pod="openshift-ingress-canary/ingress-canary-jcm4q" Sep 17 15:30:57.775344 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.775233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grfff\" (UniqueName: \"kubernetes.io/projected/78acfb4b-12cd-4266-9f36-3c13f8a18263-kube-api-access-grfff\") pod \"ingress-canary-jcm4q\" (UID: \"78acfb4b-12cd-4266-9f36-3c13f8a18263\") " pod="openshift-ingress-canary/ingress-canary-jcm4q" Sep 17 15:30:57.778855 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.778716 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78acfb4b-12cd-4266-9f36-3c13f8a18263-cert\") pod \"ingress-canary-jcm4q\" (UID: \"78acfb4b-12cd-4266-9f36-3c13f8a18263\") " pod="openshift-ingress-canary/ingress-canary-jcm4q" Sep 17 15:30:57.783376 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.783350 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grfff\" (UniqueName: \"kubernetes.io/projected/78acfb4b-12cd-4266-9f36-3c13f8a18263-kube-api-access-grfff\") pod \"ingress-canary-jcm4q\" (UID: \"78acfb4b-12cd-4266-9f36-3c13f8a18263\") " pod="openshift-ingress-canary/ingress-canary-jcm4q" Sep 17 15:30:57.801964 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.801719 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jcm4q" Sep 17 15:30:57.963869 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:57.963840 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:58.026964 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.026939 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h5587"] Sep 17 15:30:58.030267 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.030230 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65d59949f4-lb8rv"] Sep 17 15:30:58.030459 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:58.030433 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda766297c_8356_4831_815f_52186c63cd92.slice/crio-6453623e92a13e4cd5c6be6b2715d083fe99979c4bb27ecfffd273ea9d1ca978 WatchSource:0}: Error finding container 6453623e92a13e4cd5c6be6b2715d083fe99979c4bb27ecfffd273ea9d1ca978: Status 404 returned error can't find the container with id 6453623e92a13e4cd5c6be6b2715d083fe99979c4bb27ecfffd273ea9d1ca978 Sep 17 15:30:58.031672 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.031644 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:58.034277 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:58.034251 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06863d44_90d7_4d15_a6df_5597dcb22945.slice/crio-3981729ed9a56c19f3e4234a32fc4b3dde4c056e7a0cc6bf7e4e212ec938023e WatchSource:0}: Error finding container 3981729ed9a56c19f3e4234a32fc4b3dde4c056e7a0cc6bf7e4e212ec938023e: Status 404 returned error can't find the container with id 3981729ed9a56c19f3e4234a32fc4b3dde4c056e7a0cc6bf7e4e212ec938023e Sep 17 15:30:58.042472 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.042453 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ds2t5"] Sep 17 15:30:58.045030 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:58.045013 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3816cff_380c_4f26_bc82_a648050bea57.slice/crio-9b827c50c99d44c946fb0511717d5403a6d808a4260a3f2ae95c64944bad4f2a WatchSource:0}: Error finding container 9b827c50c99d44c946fb0511717d5403a6d808a4260a3f2ae95c64944bad4f2a: Status 404 returned error can't find the container with id 9b827c50c99d44c946fb0511717d5403a6d808a4260a3f2ae95c64944bad4f2a Sep 17 15:30:58.089735 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.089713 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jcm4q"] Sep 17 15:30:58.093142 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:30:58.093121 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78acfb4b_12cd_4266_9f36_3c13f8a18263.slice/crio-5655d7e8ab8a33606dc9274c2597e708f864b84d1637c7f985770bd5983bde49 WatchSource:0}: Error finding container 5655d7e8ab8a33606dc9274c2597e708f864b84d1637c7f985770bd5983bde49: Status 404 returned error can't find the container with id 5655d7e8ab8a33606dc9274c2597e708f864b84d1637c7f985770bd5983bde49 Sep 17 15:30:58.455823 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.455661 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:30:58.455823 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.455710 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:30:58.455823 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.455770 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:30:58.458561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.458535 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 15:30:58.458839 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.458815 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zrbtw\"" Sep 17 15:30:58.459306 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.459280 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 15:30:58.459390 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.459341 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 15:30:58.460797 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.460769 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 15:30:58.461390 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.461190 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pmg9m\"" Sep 17 15:30:58.861534 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.861488 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" event={"ID":"06863d44-90d7-4d15-a6df-5597dcb22945","Type":"ContainerStarted","Data":"57016dfa24eef44ea4728c2ef0122ce1beee9cfac51fb953ad9ee82eb6b87266"} Sep 17 15:30:58.862133 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.861584 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" event={"ID":"06863d44-90d7-4d15-a6df-5597dcb22945","Type":"ContainerStarted","Data":"3981729ed9a56c19f3e4234a32fc4b3dde4c056e7a0cc6bf7e4e212ec938023e"} Sep 17 15:30:58.862133 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.861700 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:30:58.863652 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.863624 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ds2t5" event={"ID":"b3816cff-380c-4f26-bc82-a648050bea57","Type":"ContainerStarted","Data":"2fcee71d5740b91ef6f25c036e9bc305c3d4a9d352b9c3344fdf8acf21dc2ec6"} Sep 17 15:30:58.863773 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.863681 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ds2t5" event={"ID":"b3816cff-380c-4f26-bc82-a648050bea57","Type":"ContainerStarted","Data":"9b827c50c99d44c946fb0511717d5403a6d808a4260a3f2ae95c64944bad4f2a"} Sep 17 15:30:58.865943 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.865897 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jcm4q" event={"ID":"78acfb4b-12cd-4266-9f36-3c13f8a18263","Type":"ContainerStarted","Data":"5655d7e8ab8a33606dc9274c2597e708f864b84d1637c7f985770bd5983bde49"} Sep 17 15:30:58.867288 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.867265 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h5587" event={"ID":"a766297c-8356-4831-815f-52186c63cd92","Type":"ContainerStarted","Data":"6453623e92a13e4cd5c6be6b2715d083fe99979c4bb27ecfffd273ea9d1ca978"} Sep 17 15:30:58.995541 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:58.995508 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:59.098685 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:59.098649 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:30:59.872263 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:30:59.872044 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ds2t5" event={"ID":"b3816cff-380c-4f26-bc82-a648050bea57","Type":"ContainerStarted","Data":"4ea9e01e9e2427f05068265d7492fce0147a91f36ed443c49554990790372e29"} Sep 17 15:31:01.082216 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.082178 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:01.185903 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.185877 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:01.257203 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.257147 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" podStartSLOduration=10.257127059 podStartE2EDuration="10.257127059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:58.88147397 +0000 UTC m=+45.877710742" watchObservedRunningTime="2026-09-17 15:31:01.257127059 +0000 UTC m=+48.253363804" Sep 17 15:31:01.257652 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.257632 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7"] Sep 17 15:31:01.260674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.260651 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:01.274716 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.274696 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7" Sep 17 15:31:01.274716 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.274707 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7"] Sep 17 15:31:01.276766 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.276745 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 17 15:31:01.276875 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.276797 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xglx2\"" Sep 17 15:31:01.300530 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.300503 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/22787295-1701-4fc1-ab50-469300195ea0-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rkhk7\" (UID: \"22787295-1701-4fc1-ab50-469300195ea0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7" Sep 17 15:31:01.401495 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.401459 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/22787295-1701-4fc1-ab50-469300195ea0-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rkhk7\" (UID: \"22787295-1701-4fc1-ab50-469300195ea0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7" Sep 17 15:31:01.401633 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:31:01.401579 2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 17 15:31:01.401686 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:31:01.401642 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22787295-1701-4fc1-ab50-469300195ea0-tls-certificates podName:22787295-1701-4fc1-ab50-469300195ea0 nodeName:}" failed. No retries permitted until 2026-09-17 15:31:01.901627026 +0000 UTC m=+48.897863763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/22787295-1701-4fc1-ab50-469300195ea0-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-rkhk7" (UID: "22787295-1701-4fc1-ab50-469300195ea0") : secret "prometheus-operator-admission-webhook-tls" not found Sep 17 15:31:01.651269 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.651191 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:01.725479 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.725450 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:01.904483 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.904455 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/22787295-1701-4fc1-ab50-469300195ea0-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rkhk7\" (UID: \"22787295-1701-4fc1-ab50-469300195ea0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7" Sep 17 15:31:01.908577 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.908536 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/22787295-1701-4fc1-ab50-469300195ea0-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rkhk7\" (UID: \"22787295-1701-4fc1-ab50-469300195ea0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7" Sep 17 15:31:01.949558 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:01.949512 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:02.044962 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.044899 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:02.183684 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.183653 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7" Sep 17 15:31:02.480519 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.480260 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7"] Sep 17 15:31:02.483372 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:31:02.483348 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22787295_1701_4fc1_ab50_469300195ea0.slice/crio-d309ed60eba39d6669e6a8e0537ebdb50eb364d6d7646791f2389c3febd9d804 WatchSource:0}: Error finding container d309ed60eba39d6669e6a8e0537ebdb50eb364d6d7646791f2389c3febd9d804: Status 404 returned error can't find the container with id d309ed60eba39d6669e6a8e0537ebdb50eb364d6d7646791f2389c3febd9d804 Sep 17 15:31:02.498311 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.498288 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:02.587798 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.587768 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:02.883981 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.883946 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jcm4q" event={"ID":"78acfb4b-12cd-4266-9f36-3c13f8a18263","Type":"ContainerStarted","Data":"8e5fd8bcb55f5b67860d06a397cb3b70b8c3a4421fe0b310a69db60641f6148f"} Sep 17 15:31:02.885664 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.885639 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h5587" event={"ID":"a766297c-8356-4831-815f-52186c63cd92","Type":"ContainerStarted","Data":"d4310f5e3b6503978a9a7a046376c5736921a5a39b5ac018ebf0181df50b8e75"} Sep 17 15:31:02.885808 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.885691 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h5587" Sep 17 15:31:02.885808 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.885715 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h5587" event={"ID":"a766297c-8356-4831-815f-52186c63cd92","Type":"ContainerStarted","Data":"88e636b084a475d91b72e0f68094f291f3cc6b0a73a34e97b4cf3d826acaaff5"} Sep 17 15:31:02.887490 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.887466 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ds2t5" event={"ID":"b3816cff-380c-4f26-bc82-a648050bea57","Type":"ContainerStarted","Data":"f92620c16cb5d2f629c90686cb00ec317a29c35dbc4976b3e121f804b8239292"} Sep 17 15:31:02.888518 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.888501 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7" event={"ID":"22787295-1701-4fc1-ab50-469300195ea0","Type":"ContainerStarted","Data":"d309ed60eba39d6669e6a8e0537ebdb50eb364d6d7646791f2389c3febd9d804"} Sep 17 15:31:02.900077 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.899952 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jcm4q" podStartSLOduration=2.172225234 podStartE2EDuration="5.899934517s" totalImagesPullingTime="3.727709283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:02.899258986 +0000 UTC m=+49.895495734" watchObservedRunningTime="2026-09-17 15:31:02.899934517 +0000 UTC m=+49.896171257" Sep 17 15:31:02.917495 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.917452 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h5587" podStartSLOduration=2.136290775 podStartE2EDuration="5.91743774s" totalImagesPullingTime="3.781146965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:02.916906392 +0000 UTC m=+49.913143137" watchObservedRunningTime="2026-09-17 15:31:02.91743774 +0000 UTC m=+49.913674487" Sep 17 15:31:02.936754 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:02.936701 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ds2t5" podStartSLOduration=2.108724362 podStartE2EDuration="5.936679281s" totalImagesPullingTime="3.827954919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:02.935823177 +0000 UTC m=+49.932059922" watchObservedRunningTime="2026-09-17 15:31:02.936679281 +0000 UTC m=+49.932916033" Sep 17 15:31:03.195783 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:03.195697 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:03.256470 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:03.256438 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:03.781309 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:03.781280 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:03.847570 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:03.847538 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:03.913060 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:03.913030 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:03.970216 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:03.970187 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:04.896342 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:04.896153 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7" event={"ID":"22787295-1701-4fc1-ab50-469300195ea0","Type":"ContainerStarted","Data":"dbd7c498e6c6dd968e6f0aecf3e6fa0e5d4a6247e0e7ba780b150bc1805e0ec0"} Sep 17 15:31:04.896739 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:04.896406 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7" Sep 17 15:31:04.901097 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:04.901074 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7" Sep 17 15:31:04.912235 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:04.912189 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rkhk7" podStartSLOduration=2.3680696 podStartE2EDuration="3.912177534s" totalImagesPullingTime="1.544107934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:04.910818332 +0000 UTC m=+51.907055079" watchObservedRunningTime="2026-09-17 15:31:04.912177534 +0000 UTC m=+51.908414280" Sep 17 15:31:05.289429 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:05.289405 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:05.364586 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:05.364548 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:05.534083 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:05.534051 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:05.787926 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:05.787888 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:05.891350 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:05.891318 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:07.206319 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.206287 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:07.311358 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.311327 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:07.404938 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.404896 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69dbdbd59-47qx5"] Sep 17 15:31:07.406608 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.406591 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.413179 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.412734 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 17 15:31:07.413179 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.413021 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 17 15:31:07.413179 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.413119 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 17 15:31:07.413616 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.413594 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 17 15:31:07.413760 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.413731 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:07.413927 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.413898 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 17 15:31:07.414485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.414470 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 17 15:31:07.414647 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.414634 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 17 15:31:07.415636 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.415619 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4tjvv\"" Sep 17 15:31:07.419983 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.419967 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 17 15:31:07.422835 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.422816 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69dbdbd59-47qx5"] Sep 17 15:31:07.446577 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.446547 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09afc480-bdfd-4b90-83c3-561898a44662-console-serving-cert\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.446693 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.446592 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d8dw\" (UniqueName: \"kubernetes.io/projected/09afc480-bdfd-4b90-83c3-561898a44662-kube-api-access-4d8dw\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.446693 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.446678 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09afc480-bdfd-4b90-83c3-561898a44662-console-oauth-config\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.446767 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.446712 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-oauth-serving-cert\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.446767 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.446731 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-trusted-ca-bundle\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.446767 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.446748 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-console-config\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.446767 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.446763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-service-ca\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.547364 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.547328 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09afc480-bdfd-4b90-83c3-561898a44662-console-oauth-config\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.547557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.547386 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-oauth-serving-cert\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.547557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.547417 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-trusted-ca-bundle\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.547557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.547444 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-console-config\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.547557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.547469 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-service-ca\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.547557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.547497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09afc480-bdfd-4b90-83c3-561898a44662-console-serving-cert\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.547557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.547540 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d8dw\" (UniqueName: \"kubernetes.io/projected/09afc480-bdfd-4b90-83c3-561898a44662-kube-api-access-4d8dw\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.548270 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.548242 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-console-config\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.548377 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.548242 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-oauth-serving-cert\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.548431 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.548392 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-service-ca\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.548758 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.548730 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-trusted-ca-bundle\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.551346 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.551326 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09afc480-bdfd-4b90-83c3-561898a44662-console-oauth-config\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.551451 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.551351 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09afc480-bdfd-4b90-83c3-561898a44662-console-serving-cert\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.555562 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.555542 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d8dw\" (UniqueName: \"kubernetes.io/projected/09afc480-bdfd-4b90-83c3-561898a44662-kube-api-access-4d8dw\") pod \"console-69dbdbd59-47qx5\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:07.566809 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.566788 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:07.667612 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.667587 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:07.716040 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:07.716011 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:08.001250 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:08.001174 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69dbdbd59-47qx5"] Sep 17 15:31:08.003896 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:31:08.003870 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09afc480_bdfd_4b90_83c3_561898a44662.slice/crio-df9931c6ca89e5e7c7330f282b1c12739dc372cb7553ab50ce01f00543ee1992 WatchSource:0}: Error finding container df9931c6ca89e5e7c7330f282b1c12739dc372cb7553ab50ce01f00543ee1992: Status 404 returned error can't find the container with id df9931c6ca89e5e7c7330f282b1c12739dc372cb7553ab50ce01f00543ee1992 Sep 17 15:31:08.904241 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:08.904140 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:08.913828 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:08.913769 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69dbdbd59-47qx5" event={"ID":"09afc480-bdfd-4b90-83c3-561898a44662","Type":"ContainerStarted","Data":"df9931c6ca89e5e7c7330f282b1c12739dc372cb7553ab50ce01f00543ee1992"} Sep 17 15:31:09.003219 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.003177 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:09.430330 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.430297 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:09.654787 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.654749 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:09.681517 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.681438 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh"] Sep 17 15:31:09.703730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.703705 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-przmb"] Sep 17 15:31:09.703880 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.703838 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:09.706340 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.706311 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 17 15:31:09.706340 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.706329 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 15:31:09.706886 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.706863 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 15:31:09.707012 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.706893 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xdgwr\"" Sep 17 15:31:09.707079 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.707008 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 17 15:31:09.707166 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.707149 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 15:31:09.717724 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.717702 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh"] Sep 17 15:31:09.717831 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.717752 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8rdxm"] Sep 17 15:31:09.717831 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.717816 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.726421 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.726403 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bjzg7\"" Sep 17 15:31:09.726519 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.726402 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 17 15:31:09.726519 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.726455 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 17 15:31:09.730120 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.730100 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 17 15:31:09.738171 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.738152 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-przmb"] Sep 17 15:31:09.738299 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.738284 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.740329 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.740312 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 15:31:09.740522 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.740507 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jw4fs\"" Sep 17 15:31:09.740771 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.740759 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 15:31:09.741192 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.741177 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 15:31:09.752975 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.752956 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:09.765163 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765137 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60318aea-c20b-411a-86ec-3aa0b94d2edc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-q7rdh\" (UID: \"60318aea-c20b-411a-86ec-3aa0b94d2edc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:09.765265 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765167 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3982a855-709a-4b05-b4ee-a2d74565b64e-metrics-client-ca\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.765265 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765188 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h529z\" (UniqueName: \"kubernetes.io/projected/3982a855-709a-4b05-b4ee-a2d74565b64e-kube-api-access-h529z\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.765265 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765241 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-textfile\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.765369 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765264 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-tls\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.765369 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.765369 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765308 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb4gl\" (UniqueName: \"kubernetes.io/projected/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-api-access-fb4gl\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.765369 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.765369 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765363 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.765523 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765415 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-wtmp\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.765523 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765439 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60318aea-c20b-411a-86ec-3aa0b94d2edc-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-q7rdh\" (UID: \"60318aea-c20b-411a-86ec-3aa0b94d2edc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:09.765523 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765467 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3982a855-709a-4b05-b4ee-a2d74565b64e-sys\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.765523 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765483 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdz6\" (UniqueName: \"kubernetes.io/projected/60318aea-c20b-411a-86ec-3aa0b94d2edc-kube-api-access-drdz6\") pod \"openshift-state-metrics-67dcf8c979-q7rdh\" (UID: \"60318aea-c20b-411a-86ec-3aa0b94d2edc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:09.765523 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765497 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3982a855-709a-4b05-b4ee-a2d74565b64e-root\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.765667 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765527 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66629cca-8fa3-4a89-97ec-dc73a66d9022-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.765667 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765543 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60318aea-c20b-411a-86ec-3aa0b94d2edc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-q7rdh\" (UID: \"60318aea-c20b-411a-86ec-3aa0b94d2edc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:09.765667 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765580 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/66629cca-8fa3-4a89-97ec-dc73a66d9022-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.765667 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765618 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.765667 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.765639 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.866303 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866262 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.866303 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866308 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.866640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866350 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-wtmp\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.866640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866378 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60318aea-c20b-411a-86ec-3aa0b94d2edc-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-q7rdh\" (UID: \"60318aea-c20b-411a-86ec-3aa0b94d2edc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:09.866640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866405 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3982a855-709a-4b05-b4ee-a2d74565b64e-sys\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.866640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866429 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdz6\" (UniqueName: \"kubernetes.io/projected/60318aea-c20b-411a-86ec-3aa0b94d2edc-kube-api-access-drdz6\") pod \"openshift-state-metrics-67dcf8c979-q7rdh\" (UID: \"60318aea-c20b-411a-86ec-3aa0b94d2edc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:09.866640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866450 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3982a855-709a-4b05-b4ee-a2d74565b64e-root\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.866640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866489 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66629cca-8fa3-4a89-97ec-dc73a66d9022-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.866640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866512 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60318aea-c20b-411a-86ec-3aa0b94d2edc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-q7rdh\" (UID: \"60318aea-c20b-411a-86ec-3aa0b94d2edc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:09.866640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866536 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/66629cca-8fa3-4a89-97ec-dc73a66d9022-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.866640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866578 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3982a855-709a-4b05-b4ee-a2d74565b64e-sys\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.867150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866676 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-wtmp\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.867150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866703 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.867150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866871 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/66629cca-8fa3-4a89-97ec-dc73a66d9022-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.867150 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:31:09.866774 2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 17 15:31:09.867150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866950 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.867150 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:31:09.866982 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-tls podName:66629cca-8fa3-4a89-97ec-dc73a66d9022 nodeName:}" failed. No retries permitted until 2026-09-17 15:31:10.366961621 +0000 UTC m=+57.363198345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-przmb" (UID: "66629cca-8fa3-4a89-97ec-dc73a66d9022") : secret "kube-state-metrics-tls" not found Sep 17 15:31:09.867150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866995 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.867150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.867027 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60318aea-c20b-411a-86ec-3aa0b94d2edc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-q7rdh\" (UID: \"60318aea-c20b-411a-86ec-3aa0b94d2edc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:09.867150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.867062 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3982a855-709a-4b05-b4ee-a2d74565b64e-metrics-client-ca\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.867150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.867094 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h529z\" (UniqueName: \"kubernetes.io/projected/3982a855-709a-4b05-b4ee-a2d74565b64e-kube-api-access-h529z\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.867150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.867136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-textfile\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.867150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.867158 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-tls\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.867744 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.867180 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.867744 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.867218 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb4gl\" (UniqueName: \"kubernetes.io/projected/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-api-access-fb4gl\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.867744 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.867430 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66629cca-8fa3-4a89-97ec-dc73a66d9022-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.867744 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:31:09.867559 2581 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 17 15:31:09.867744 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.867587 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60318aea-c20b-411a-86ec-3aa0b94d2edc-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-q7rdh\" (UID: \"60318aea-c20b-411a-86ec-3aa0b94d2edc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:09.867744 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:31:09.867604 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60318aea-c20b-411a-86ec-3aa0b94d2edc-openshift-state-metrics-tls podName:60318aea-c20b-411a-86ec-3aa0b94d2edc nodeName:}" failed. No retries permitted until 2026-09-17 15:31:10.36759067 +0000 UTC m=+57.363827412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/60318aea-c20b-411a-86ec-3aa0b94d2edc-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-q7rdh" (UID: "60318aea-c20b-411a-86ec-3aa0b94d2edc") : secret "openshift-state-metrics-tls" not found Sep 17 15:31:09.867744 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.867588 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:09.867744 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.867671 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.867744 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.866768 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3982a855-709a-4b05-b4ee-a2d74565b64e-root\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.868210 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.868045 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3982a855-709a-4b05-b4ee-a2d74565b64e-metrics-client-ca\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.868312 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.868219 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-textfile\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.869443 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.869412 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.870195 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.870168 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-tls\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.870479 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.870451 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60318aea-c20b-411a-86ec-3aa0b94d2edc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-q7rdh\" (UID: \"60318aea-c20b-411a-86ec-3aa0b94d2edc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:09.870609 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.870588 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3982a855-709a-4b05-b4ee-a2d74565b64e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.886654 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.886624 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdz6\" (UniqueName: \"kubernetes.io/projected/60318aea-c20b-411a-86ec-3aa0b94d2edc-kube-api-access-drdz6\") pod \"openshift-state-metrics-67dcf8c979-q7rdh\" (UID: \"60318aea-c20b-411a-86ec-3aa0b94d2edc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:09.887795 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.887769 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h529z\" (UniqueName: \"kubernetes.io/projected/3982a855-709a-4b05-b4ee-a2d74565b64e-kube-api-access-h529z\") pod \"node-exporter-8rdxm\" (UID: \"3982a855-709a-4b05-b4ee-a2d74565b64e\") " pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:09.888768 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.888744 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb4gl\" (UniqueName: \"kubernetes.io/projected/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-api-access-fb4gl\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:09.964471 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:09.964396 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:10.046678 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:10.046647 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8rdxm" Sep 17 15:31:10.068342 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:10.068307 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:10.190991 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:10.190964 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:10.289028 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:10.288995 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:10.372638 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:10.372601 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60318aea-c20b-411a-86ec-3aa0b94d2edc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-q7rdh\" (UID: \"60318aea-c20b-411a-86ec-3aa0b94d2edc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:10.372809 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:10.372650 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:10.372882 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:31:10.372801 2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 17 15:31:10.372950 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:31:10.372884 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-tls podName:66629cca-8fa3-4a89-97ec-dc73a66d9022 nodeName:}" failed. No retries permitted until 2026-09-17 15:31:11.372861051 +0000 UTC m=+58.369097776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-przmb" (UID: "66629cca-8fa3-4a89-97ec-dc73a66d9022") : secret "kube-state-metrics-tls" not found Sep 17 15:31:10.375385 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:10.375366 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/60318aea-c20b-411a-86ec-3aa0b94d2edc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-q7rdh\" (UID: \"60318aea-c20b-411a-86ec-3aa0b94d2edc\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:10.428447 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:10.428412 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:10.526674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:10.526642 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:10.614977 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:10.614884 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" Sep 17 15:31:10.628488 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:10.628457 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:10.728850 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:10.728815 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:10.966752 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:10.966682 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:10.971598 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:31:10.971572 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3982a855_709a_4b05_b4ee_a2d74565b64e.slice/crio-a31c5f71d911c9ab94abc201d14e7dee249d5502cdf25e37edfc4399cd2878cf WatchSource:0}: Error finding container a31c5f71d911c9ab94abc201d14e7dee249d5502cdf25e37edfc4399cd2878cf: Status 404 returned error can't find the container with id a31c5f71d911c9ab94abc201d14e7dee249d5502cdf25e37edfc4399cd2878cf Sep 17 15:31:11.203824 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.203794 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:11.246433 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.246363 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh"] Sep 17 15:31:11.251822 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:31:11.251796 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60318aea_c20b_411a_86ec_3aa0b94d2edc.slice/crio-c9ab1af2c7da8364c1ea8f02b7a74be37aeb898615695f028abaef564c036272 WatchSource:0}: Error finding container c9ab1af2c7da8364c1ea8f02b7a74be37aeb898615695f028abaef564c036272: Status 404 returned error can't find the container with id c9ab1af2c7da8364c1ea8f02b7a74be37aeb898615695f028abaef564c036272 Sep 17 15:31:11.379282 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.379248 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:11.381815 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.381783 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/66629cca-8fa3-4a89-97ec-dc73a66d9022-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-przmb\" (UID: \"66629cca-8fa3-4a89-97ec-dc73a66d9022\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:11.526886 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.526857 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" Sep 17 15:31:11.748166 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.748126 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:11.804678 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.804607 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-przmb"] Sep 17 15:31:11.810590 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:31:11.810567 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66629cca_8fa3_4a89_97ec_dc73a66d9022.slice/crio-1accf02be2616b2ccbde086aa858808f3eada0a8aec79293b9761e34352fc52e WatchSource:0}: Error finding container 1accf02be2616b2ccbde086aa858808f3eada0a8aec79293b9761e34352fc52e: Status 404 returned error can't find the container with id 1accf02be2616b2ccbde086aa858808f3eada0a8aec79293b9761e34352fc52e Sep 17 15:31:11.926355 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.926092 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69dbdbd59-47qx5" event={"ID":"09afc480-bdfd-4b90-83c3-561898a44662","Type":"ContainerStarted","Data":"8bb8b312b5dff25442cd884830da34612220688eafa7ad3fad3dfd9baa355dfd"} Sep 17 15:31:11.928133 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.928094 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" event={"ID":"66629cca-8fa3-4a89-97ec-dc73a66d9022","Type":"ContainerStarted","Data":"1accf02be2616b2ccbde086aa858808f3eada0a8aec79293b9761e34352fc52e"} Sep 17 15:31:11.931001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.930878 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" event={"ID":"60318aea-c20b-411a-86ec-3aa0b94d2edc","Type":"ContainerStarted","Data":"7e9f19c8da04cde409ee1f9fc94c5138a5a925f94ebdadcb7310f6874998b540"} Sep 17 15:31:11.931001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.930958 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" event={"ID":"60318aea-c20b-411a-86ec-3aa0b94d2edc","Type":"ContainerStarted","Data":"9a02a9f131cd30874f4da25734e107a4c2de2cfa7942c2b9211a8b3ad34d3f4c"} Sep 17 15:31:11.931001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.930980 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" event={"ID":"60318aea-c20b-411a-86ec-3aa0b94d2edc","Type":"ContainerStarted","Data":"c9ab1af2c7da8364c1ea8f02b7a74be37aeb898615695f028abaef564c036272"} Sep 17 15:31:11.932975 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.932938 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rdxm" event={"ID":"3982a855-709a-4b05-b4ee-a2d74565b64e","Type":"ContainerStarted","Data":"b5d2ae8618f9d6a2d555c68a47251d6f1a3ff6c42df63367d43d491d02c363a5"} Sep 17 15:31:11.933073 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.932986 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rdxm" event={"ID":"3982a855-709a-4b05-b4ee-a2d74565b64e","Type":"ContainerStarted","Data":"a31c5f71d911c9ab94abc201d14e7dee249d5502cdf25e37edfc4399cd2878cf"} Sep 17 15:31:11.944183 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:11.944145 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69dbdbd59-47qx5" podStartSLOduration=1.688735758 podStartE2EDuration="4.944131187s" totalImagesPullingTime="3.255395429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:11.943106221 +0000 UTC m=+58.939342968" watchObservedRunningTime="2026-09-17 15:31:11.944131187 +0000 UTC m=+58.940367977" Sep 17 15:31:12.657513 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:12.657425 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:12.737984 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:12.737894 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:12.816132 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:12.816108 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cvwgv" Sep 17 15:31:12.896140 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:12.896116 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h5587" Sep 17 15:31:12.939958 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:12.939836 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" event={"ID":"60318aea-c20b-411a-86ec-3aa0b94d2edc","Type":"ContainerStarted","Data":"d17166b141dda6e786e6ce2eda2c6c30ef6312f02578335de15be76209b42881"} Sep 17 15:31:12.942174 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:12.942142 2581 generic.go:356] "Generic (PLEG): container finished" podID="3982a855-709a-4b05-b4ee-a2d74565b64e" containerID="b5d2ae8618f9d6a2d555c68a47251d6f1a3ff6c42df63367d43d491d02c363a5" exitCode=0 Sep 17 15:31:12.942323 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:12.942204 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rdxm" event={"ID":"3982a855-709a-4b05-b4ee-a2d74565b64e","Type":"ContainerDied","Data":"b5d2ae8618f9d6a2d555c68a47251d6f1a3ff6c42df63367d43d491d02c363a5"} Sep 17 15:31:12.965232 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:12.965189 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-q7rdh" podStartSLOduration=2.982874395 podStartE2EDuration="3.965171598s" totalImagesPullingTime="982.297203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:12.964132245 +0000 UTC m=+59.960368992" watchObservedRunningTime="2026-09-17 15:31:12.965171598 +0000 UTC m=+59.961408343" Sep 17 15:31:12.991972 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:12.991936 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:13.082624 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:13.082594 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:13.182714 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:13.182694 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:13.285681 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:13.285653 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:13.949210 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:13.949173 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" event={"ID":"66629cca-8fa3-4a89-97ec-dc73a66d9022","Type":"ContainerStarted","Data":"337ea2e88c471d2cc34ce57e2bcf35c54efb5e1907e7d0da856bc04a59d9ac3a"} Sep 17 15:31:13.949654 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:13.949228 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" event={"ID":"66629cca-8fa3-4a89-97ec-dc73a66d9022","Type":"ContainerStarted","Data":"cc74ea52fa4defb9728e45826ba595e40484715783991528cde174a59b12b8df"} Sep 17 15:31:13.949654 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:13.949256 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" event={"ID":"66629cca-8fa3-4a89-97ec-dc73a66d9022","Type":"ContainerStarted","Data":"a75874102d9d824579f04e9231eb6bd5d17c57f39c1f845eedee07726eab3cd9"} Sep 17 15:31:13.951295 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:13.951272 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rdxm" event={"ID":"3982a855-709a-4b05-b4ee-a2d74565b64e","Type":"ContainerStarted","Data":"8262af1c33e21780c56e467e6d555bc3a8001d5a713d0fc5822bf88841682af1"} Sep 17 15:31:13.951380 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:13.951321 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rdxm" event={"ID":"3982a855-709a-4b05-b4ee-a2d74565b64e","Type":"ContainerStarted","Data":"904cc6a7562bd2924b9a34ce53ba73481d47b4a730d34c380608020c46de079a"} Sep 17 15:31:13.968373 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:13.968325 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-przmb" podStartSLOduration=3.628556176 podStartE2EDuration="4.96829665s" totalImagesPullingTime="1.339740474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:13.967284362 +0000 UTC m=+60.963521111" watchObservedRunningTime="2026-09-17 15:31:13.96829665 +0000 UTC m=+60.964533397" Sep 17 15:31:13.995697 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:13.995645 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8rdxm" podStartSLOduration=4.084862285 podStartE2EDuration="4.995523862s" totalImagesPullingTime="841.674911ms" totalInitContainerRuntime="68.986666ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:13.994965929 +0000 UTC m=+60.991202675" watchObservedRunningTime="2026-09-17 15:31:13.995523862 +0000 UTC m=+60.991760608" Sep 17 15:31:14.491327 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:14.491294 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:14.586423 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:14.586388 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:14.978803 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:14.978775 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-74dd668c7f-gpk2h"] Sep 17 15:31:14.981146 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:14.981128 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:14.983387 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:14.983363 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 17 15:31:14.983519 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:14.983401 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 17 15:31:14.983519 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:14.983412 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-n6nt2\"" Sep 17 15:31:14.983519 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:14.983472 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 17 15:31:14.983725 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:14.983712 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 17 15:31:14.984016 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:14.983993 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 17 15:31:14.989734 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:14.989717 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 17 15:31:15.001596 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.001572 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74dd668c7f-gpk2h"] Sep 17 15:31:15.010635 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.010609 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/36d804e1-cd53-437b-941c-4adfa0d28648-federate-client-tls\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.010717 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.010638 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/36d804e1-cd53-437b-941c-4adfa0d28648-telemeter-client-tls\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.010717 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.010655 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36d804e1-cd53-437b-941c-4adfa0d28648-metrics-client-ca\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.010717 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.010670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxtq\" (UniqueName: \"kubernetes.io/projected/36d804e1-cd53-437b-941c-4adfa0d28648-kube-api-access-ssxtq\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.010717 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.010715 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36d804e1-cd53-437b-941c-4adfa0d28648-serving-certs-ca-bundle\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.010939 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.010906 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/36d804e1-cd53-437b-941c-4adfa0d28648-secret-telemeter-client\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.011044 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.011020 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36d804e1-cd53-437b-941c-4adfa0d28648-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.011124 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.011086 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36d804e1-cd53-437b-941c-4adfa0d28648-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.111392 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.111359 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36d804e1-cd53-437b-941c-4adfa0d28648-serving-certs-ca-bundle\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.111518 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.111414 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/36d804e1-cd53-437b-941c-4adfa0d28648-secret-telemeter-client\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.111583 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.111558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36d804e1-cd53-437b-941c-4adfa0d28648-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.111640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.111621 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36d804e1-cd53-437b-941c-4adfa0d28648-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.111698 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.111663 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/36d804e1-cd53-437b-941c-4adfa0d28648-federate-client-tls\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.111752 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.111702 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/36d804e1-cd53-437b-941c-4adfa0d28648-telemeter-client-tls\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.111752 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.111724 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36d804e1-cd53-437b-941c-4adfa0d28648-metrics-client-ca\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.111847 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.111749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxtq\" (UniqueName: \"kubernetes.io/projected/36d804e1-cd53-437b-941c-4adfa0d28648-kube-api-access-ssxtq\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.112201 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.112171 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36d804e1-cd53-437b-941c-4adfa0d28648-serving-certs-ca-bundle\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.112511 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.112488 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36d804e1-cd53-437b-941c-4adfa0d28648-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.112613 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.112495 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36d804e1-cd53-437b-941c-4adfa0d28648-metrics-client-ca\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.114018 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.113988 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/36d804e1-cd53-437b-941c-4adfa0d28648-secret-telemeter-client\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.114129 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.114110 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/36d804e1-cd53-437b-941c-4adfa0d28648-federate-client-tls\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.114129 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.114122 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/36d804e1-cd53-437b-941c-4adfa0d28648-telemeter-client-tls\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.114251 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.114226 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36d804e1-cd53-437b-941c-4adfa0d28648-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.119730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.119705 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxtq\" (UniqueName: \"kubernetes.io/projected/36d804e1-cd53-437b-941c-4adfa0d28648-kube-api-access-ssxtq\") pod \"telemeter-client-74dd668c7f-gpk2h\" (UID: \"36d804e1-cd53-437b-941c-4adfa0d28648\") " pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.289928 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.289881 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" Sep 17 15:31:15.547717 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.547651 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:15.571557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.571521 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74dd668c7f-gpk2h"] Sep 17 15:31:15.574037 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:31:15.574010 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d804e1_cd53_437b_941c_4adfa0d28648.slice/crio-f5af08d6ffcba3bae86ac691a79f4b4b8e296ab496b6f043cc9b7234d9ef24b3 WatchSource:0}: Error finding container f5af08d6ffcba3bae86ac691a79f4b4b8e296ab496b6f043cc9b7234d9ef24b3: Status 404 returned error can't find the container with id f5af08d6ffcba3bae86ac691a79f4b4b8e296ab496b6f043cc9b7234d9ef24b3 Sep 17 15:31:15.960350 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:15.960260 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" event={"ID":"36d804e1-cd53-437b-941c-4adfa0d28648","Type":"ContainerStarted","Data":"f5af08d6ffcba3bae86ac691a79f4b4b8e296ab496b6f043cc9b7234d9ef24b3"} Sep 17 15:31:16.006331 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.006300 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:16.200561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.200525 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69dbdbd59-47qx5"] Sep 17 15:31:16.235940 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.235836 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bb78f6c45-5x666"] Sep 17 15:31:16.239730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.239706 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.258544 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.258504 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb78f6c45-5x666"] Sep 17 15:31:16.322449 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.322417 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-oauth-serving-cert\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.322629 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.322461 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-serving-cert\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.322629 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.322496 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6vlv\" (UniqueName: \"kubernetes.io/projected/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-kube-api-access-j6vlv\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.322629 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.322607 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-oauth-config\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.322749 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.322696 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-trusted-ca-bundle\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.322797 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.322765 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-config\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.322797 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.322794 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-service-ca\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.423895 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.423861 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-service-ca\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.424066 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.423958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-oauth-serving-cert\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.424066 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.423980 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-serving-cert\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.424066 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.424001 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6vlv\" (UniqueName: \"kubernetes.io/projected/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-kube-api-access-j6vlv\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.424066 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.424018 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-oauth-config\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.424066 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.424043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-trusted-ca-bundle\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.424354 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.424082 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-config\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.424721 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.424698 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-oauth-serving-cert\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.424721 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.424716 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-service-ca\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.424866 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.424851 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-config\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.425069 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.425054 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-trusted-ca-bundle\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.426521 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.426494 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-oauth-config\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.426626 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.426549 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-serving-cert\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.432208 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.432185 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6vlv\" (UniqueName: \"kubernetes.io/projected/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-kube-api-access-j6vlv\") pod \"console-bb78f6c45-5x666\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.548962 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.548928 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:16.825032 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.824960 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb78f6c45-5x666"] Sep 17 15:31:16.829278 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:31:16.829253 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5bd915_0ea2_446f_b5e8_80be6a3e1a9d.slice/crio-7f8e059ca7d5ff17d9cf08c9ed7d2faa46045565073e2c33b94584b70955ed87 WatchSource:0}: Error finding container 7f8e059ca7d5ff17d9cf08c9ed7d2faa46045565073e2c33b94584b70955ed87: Status 404 returned error can't find the container with id 7f8e059ca7d5ff17d9cf08c9ed7d2faa46045565073e2c33b94584b70955ed87 Sep 17 15:31:16.965606 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.965571 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb78f6c45-5x666" event={"ID":"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d","Type":"ContainerStarted","Data":"47c98b014b42e65a80fc9e07eab66901d53ddce7dd005424afeca9dc1a558592"} Sep 17 15:31:16.965778 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.965629 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb78f6c45-5x666" event={"ID":"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d","Type":"ContainerStarted","Data":"7f8e059ca7d5ff17d9cf08c9ed7d2faa46045565073e2c33b94584b70955ed87"} Sep 17 15:31:16.985614 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:16.985559 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bb78f6c45-5x666" podStartSLOduration=0.985538744 podStartE2EDuration="985.538744ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:16.985143735 +0000 UTC m=+63.981380484" watchObservedRunningTime="2026-09-17 15:31:16.985538744 +0000 UTC m=+63.981775487" Sep 17 15:31:17.716203 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:17.716171 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:17.886232 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:17.886198 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:17.984027 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:17.983956 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:18.085794 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:18.085757 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:18.626140 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:18.626106 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:18.733654 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:18.733580 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:18.975475 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:18.975441 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" event={"ID":"36d804e1-cd53-437b-941c-4adfa0d28648","Type":"ContainerStarted","Data":"28f40306e5462edb5c9af3d4b07f8c0f5d4b46a1a2cdba7fc3e7bf59e7a217ba"} Sep 17 15:31:19.019120 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.019043 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:19.117331 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.117288 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:19.147582 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.147550 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:31:19.151392 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.151373 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 15:31:19.160368 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.160345 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/721e805e-1d0a-473c-be1a-871d62cfebbe-original-pull-secret\") pod \"global-pull-secret-syncer-5rf6m\" (UID: \"721e805e-1d0a-473c-be1a-871d62cfebbe\") " pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:31:19.168289 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.168264 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5rf6m" Sep 17 15:31:19.201171 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.201137 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:19.247984 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.247957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs\") pod \"network-metrics-daemon-cgcpl\" (UID: \"fa257b5d-59d6-4286-a99f-b2bee16de288\") " pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:31:19.250275 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.250249 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 15:31:19.261888 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.261865 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa257b5d-59d6-4286-a99f-b2bee16de288-metrics-certs\") pod \"network-metrics-daemon-cgcpl\" (UID: \"fa257b5d-59d6-4286-a99f-b2bee16de288\") " pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:31:19.348656 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.348618 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8fzq\" (UniqueName: \"kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq\") pod \"network-check-target-6mlp5\" (UID: \"af87ecf6-e9d9-4c65-ace2-bc1709237610\") " pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:31:19.351135 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.351106 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 15:31:19.361220 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.361198 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 15:31:19.371838 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.371815 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8fzq\" (UniqueName: \"kubernetes.io/projected/af87ecf6-e9d9-4c65-ace2-bc1709237610-kube-api-access-h8fzq\") pod \"network-check-target-6mlp5\" (UID: \"af87ecf6-e9d9-4c65-ace2-bc1709237610\") " pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:31:19.479122 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.479101 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pmg9m\"" Sep 17 15:31:19.484070 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.484051 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zrbtw\"" Sep 17 15:31:19.486895 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.486876 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cgcpl" Sep 17 15:31:19.492709 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.492692 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:31:19.646071 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.646004 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5rf6m"] Sep 17 15:31:19.648722 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:31:19.648681 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721e805e_1d0a_473c_be1a_871d62cfebbe.slice/crio-a069040e41308ff0bc9bd73c032bdcdba3c2bd0a1d8377822709aa09c64d7b22 WatchSource:0}: Error finding container a069040e41308ff0bc9bd73c032bdcdba3c2bd0a1d8377822709aa09c64d7b22: Status 404 returned error can't find the container with id a069040e41308ff0bc9bd73c032bdcdba3c2bd0a1d8377822709aa09c64d7b22 Sep 17 15:31:19.763828 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.763783 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cgcpl"] Sep 17 15:31:19.765775 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:31:19.765739 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa257b5d_59d6_4286_a99f_b2bee16de288.slice/crio-22a47469385e83ec31411eb1e2244a36da4929e413926c5748604f215faae64d WatchSource:0}: Error finding container 22a47469385e83ec31411eb1e2244a36da4929e413926c5748604f215faae64d: Status 404 returned error can't find the container with id 22a47469385e83ec31411eb1e2244a36da4929e413926c5748604f215faae64d Sep 17 15:31:19.781274 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.781254 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6mlp5"] Sep 17 15:31:19.783747 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:31:19.783725 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf87ecf6_e9d9_4c65_ace2_bc1709237610.slice/crio-b84fb28d410d1ec699ca18fc89a8112fac4c85511951f9e302cb02940dda6c9e WatchSource:0}: Error finding container b84fb28d410d1ec699ca18fc89a8112fac4c85511951f9e302cb02940dda6c9e: Status 404 returned error can't find the container with id b84fb28d410d1ec699ca18fc89a8112fac4c85511951f9e302cb02940dda6c9e Sep 17 15:31:19.876443 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.876423 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65d59949f4-lb8rv" Sep 17 15:31:19.980980 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.980886 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" event={"ID":"36d804e1-cd53-437b-941c-4adfa0d28648","Type":"ContainerStarted","Data":"7e09d2fdc04766c51bc929fc49dcc02a0e996e188f198fd216e0c2b880e0b258"} Sep 17 15:31:19.980980 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.980961 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" event={"ID":"36d804e1-cd53-437b-941c-4adfa0d28648","Type":"ContainerStarted","Data":"b0a436214e5948dce5d7e6955ad75976875bb3046d0db69ba3a9c0899b3585a7"} Sep 17 15:31:19.982063 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.982037 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cgcpl" event={"ID":"fa257b5d-59d6-4286-a99f-b2bee16de288","Type":"ContainerStarted","Data":"22a47469385e83ec31411eb1e2244a36da4929e413926c5748604f215faae64d"} Sep 17 15:31:19.983102 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.983079 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5rf6m" event={"ID":"721e805e-1d0a-473c-be1a-871d62cfebbe","Type":"ContainerStarted","Data":"a069040e41308ff0bc9bd73c032bdcdba3c2bd0a1d8377822709aa09c64d7b22"} Sep 17 15:31:19.984061 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:19.984041 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6mlp5" event={"ID":"af87ecf6-e9d9-4c65-ace2-bc1709237610","Type":"ContainerStarted","Data":"b84fb28d410d1ec699ca18fc89a8112fac4c85511951f9e302cb02940dda6c9e"} Sep 17 15:31:20.005945 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.005881 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-74dd668c7f-gpk2h" podStartSLOduration=2.21325848 podStartE2EDuration="6.005867628s" totalImagesPullingTime="3.792609148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:31:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:20.004199592 +0000 UTC m=+67.000436338" watchObservedRunningTime="2026-09-17 15:31:20.005867628 +0000 UTC m=+67.002104373" Sep 17 15:31:20.025215 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.025187 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:20.129630 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.129598 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:20.224607 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.224562 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb78f6c45-5x666"] Sep 17 15:31:20.234274 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.234205 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:20.261829 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.261798 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f5db856bc-x8hgh"] Sep 17 15:31:20.265599 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.265082 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.281850 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.281824 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5db856bc-x8hgh"] Sep 17 15:31:20.358962 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.358899 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-oauth-config\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.359127 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.359059 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-trusted-ca-bundle\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.359127 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.359108 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7glxn\" (UniqueName: \"kubernetes.io/projected/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-kube-api-access-7glxn\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.359236 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.359153 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-serving-cert\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.359236 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.359182 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-service-ca\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.359236 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.359212 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-oauth-serving-cert\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.359386 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.359238 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-config\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.460614 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.460579 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7glxn\" (UniqueName: \"kubernetes.io/projected/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-kube-api-access-7glxn\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.460780 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.460640 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-serving-cert\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.460780 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.460666 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-service-ca\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.460780 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.460682 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-oauth-serving-cert\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.460780 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.460697 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-config\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.460780 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.460732 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-oauth-config\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.461053 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.460800 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-trusted-ca-bundle\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.461537 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.461484 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-service-ca\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.461537 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.461525 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-oauth-serving-cert\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.461709 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.461525 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-config\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.461766 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.461727 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-trusted-ca-bundle\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.465085 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.465058 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-serving-cert\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.466507 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.466405 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-oauth-config\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.471244 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.471221 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7glxn\" (UniqueName: \"kubernetes.io/projected/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-kube-api-access-7glxn\") pod \"console-6f5db856bc-x8hgh\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.579044 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.579003 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:20.876737 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.876708 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:20.991435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:20.991408 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cgcpl" event={"ID":"fa257b5d-59d6-4286-a99f-b2bee16de288","Type":"ContainerStarted","Data":"35e3049adf55ab26186e8dde86bd10af242c783dca813217d092d4fe744e3509"} Sep 17 15:31:21.095750 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:21.095623 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5db856bc-x8hgh"] Sep 17 15:31:21.099543 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:31:21.099360 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdda5b9_6e31_4e66_9ec7_86f1920f82a6.slice/crio-84d2afbf6645113a9d33621d86d273862ac21bd233f517856b51d59f666aaa7b WatchSource:0}: Error finding container 84d2afbf6645113a9d33621d86d273862ac21bd233f517856b51d59f666aaa7b: Status 404 returned error can't find the container with id 84d2afbf6645113a9d33621d86d273862ac21bd233f517856b51d59f666aaa7b Sep 17 15:31:21.998572 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:21.998533 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cgcpl" event={"ID":"fa257b5d-59d6-4286-a99f-b2bee16de288","Type":"ContainerStarted","Data":"af5072fbb3f7b74a2fedbb2d2a7d1e9872f72f680af8d4c53fb6cb4b6d68cfbc"} Sep 17 15:31:22.000751 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:22.000726 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5db856bc-x8hgh" event={"ID":"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6","Type":"ContainerStarted","Data":"32379253f9fff78139eae5ecb989c4e89160415fea16350931218f2f08c87520"} Sep 17 15:31:22.000867 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:22.000768 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5db856bc-x8hgh" event={"ID":"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6","Type":"ContainerStarted","Data":"84d2afbf6645113a9d33621d86d273862ac21bd233f517856b51d59f666aaa7b"} Sep 17 15:31:22.017804 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:22.017759 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cgcpl" podStartSLOduration=66.94378965 podStartE2EDuration="1m8.017744138s" totalImagesPullingTime="1.073954488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:22.015986055 +0000 UTC m=+69.012222792" watchObservedRunningTime="2026-09-17 15:31:22.017744138 +0000 UTC m=+69.013980878" Sep 17 15:31:22.036098 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:22.036029 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f5db856bc-x8hgh" podStartSLOduration=2.036008917 podStartE2EDuration="2.036008917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:22.033431817 +0000 UTC m=+69.029668565" watchObservedRunningTime="2026-09-17 15:31:22.036008917 +0000 UTC m=+69.032245659" Sep 17 15:31:22.117805 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:22.117732 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:22.172271 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:22.172238 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:23.132946 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:23.132887 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:23.233763 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:23.233716 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:25.015545 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:25.015505 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5rf6m" event={"ID":"721e805e-1d0a-473c-be1a-871d62cfebbe","Type":"ContainerStarted","Data":"25a6414d701d4f26745051069e3aa7099c318a9d0b707c3f07c1f616c45259f6"} Sep 17 15:31:25.016992 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:25.016968 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6mlp5" event={"ID":"af87ecf6-e9d9-4c65-ace2-bc1709237610","Type":"ContainerStarted","Data":"6cd0986a585a81764dc47fafc785cb034b0b84c1bd4c6267ffc1f0f44d42ac35"} Sep 17 15:31:25.017091 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:25.017021 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:31:25.031311 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:25.031265 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5rf6m" podStartSLOduration=66.231721231 podStartE2EDuration="1m11.031251901s" totalImagesPullingTime="4.79953067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:25.030931934 +0000 UTC m=+72.027168675" watchObservedRunningTime="2026-09-17 15:31:25.031251901 +0000 UTC m=+72.027488647" Sep 17 15:31:25.049277 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:25.049234 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6mlp5" podStartSLOduration=66.387827413 podStartE2EDuration="1m11.049220628s" totalImagesPullingTime="4.661393215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:25.048457059 +0000 UTC m=+72.044693805" watchObservedRunningTime="2026-09-17 15:31:25.049220628 +0000 UTC m=+72.045457373" Sep 17 15:31:26.550026 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:26.549990 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:29.939907 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:29.939874 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:30.023555 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:30.023524 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:30.579535 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:30.579496 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:30.579703 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:30.579567 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:30.584221 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:30.584198 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:31.042985 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:31.042960 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:31:31.208931 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:31.208891 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:31.309488 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:31.309417 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:31.369485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:31.369459 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:32.879676 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:32.879645 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:32.939669 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:32.939644 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:41.219271 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.219129 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69dbdbd59-47qx5" podUID="09afc480-bdfd-4b90-83c3-561898a44662" containerName="console" containerID="cri-o://8bb8b312b5dff25442cd884830da34612220688eafa7ad3fad3dfd9baa355dfd" gracePeriod=15 Sep 17 15:31:41.244654 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.244631 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:41.346100 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.346073 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:41.403700 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.403678 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:41.449662 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.449636 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:41.531000 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.530973 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-console-config\") pod \"09afc480-bdfd-4b90-83c3-561898a44662\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " Sep 17 15:31:41.531155 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.531049 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-trusted-ca-bundle\") pod \"09afc480-bdfd-4b90-83c3-561898a44662\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " Sep 17 15:31:41.531155 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.531077 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09afc480-bdfd-4b90-83c3-561898a44662-console-serving-cert\") pod \"09afc480-bdfd-4b90-83c3-561898a44662\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " Sep 17 15:31:41.531155 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.531109 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-service-ca\") pod \"09afc480-bdfd-4b90-83c3-561898a44662\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " Sep 17 15:31:41.531155 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.531143 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09afc480-bdfd-4b90-83c3-561898a44662-console-oauth-config\") pod \"09afc480-bdfd-4b90-83c3-561898a44662\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " Sep 17 15:31:41.531380 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.531288 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-oauth-serving-cert\") pod \"09afc480-bdfd-4b90-83c3-561898a44662\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " Sep 17 15:31:41.531380 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.531343 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d8dw\" (UniqueName: \"kubernetes.io/projected/09afc480-bdfd-4b90-83c3-561898a44662-kube-api-access-4d8dw\") pod \"09afc480-bdfd-4b90-83c3-561898a44662\" (UID: \"09afc480-bdfd-4b90-83c3-561898a44662\") " Sep 17 15:31:41.531485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.531456 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-console-config" (OuterVolumeSpecName: "console-config") pod "09afc480-bdfd-4b90-83c3-561898a44662" (UID: "09afc480-bdfd-4b90-83c3-561898a44662"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:31:41.531544 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.531519 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-service-ca" (OuterVolumeSpecName: "service-ca") pod "09afc480-bdfd-4b90-83c3-561898a44662" (UID: "09afc480-bdfd-4b90-83c3-561898a44662"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:31:41.531594 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.531555 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "09afc480-bdfd-4b90-83c3-561898a44662" (UID: "09afc480-bdfd-4b90-83c3-561898a44662"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:31:41.531674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.531644 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "09afc480-bdfd-4b90-83c3-561898a44662" (UID: "09afc480-bdfd-4b90-83c3-561898a44662"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:31:41.531714 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.531655 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-console-config\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:41.531714 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.531698 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-trusted-ca-bundle\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:41.531778 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.531714 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:41.533440 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.533418 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09afc480-bdfd-4b90-83c3-561898a44662-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "09afc480-bdfd-4b90-83c3-561898a44662" (UID: "09afc480-bdfd-4b90-83c3-561898a44662"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:31:41.533521 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.533499 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09afc480-bdfd-4b90-83c3-561898a44662-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "09afc480-bdfd-4b90-83c3-561898a44662" (UID: "09afc480-bdfd-4b90-83c3-561898a44662"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:31:41.533564 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.533535 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09afc480-bdfd-4b90-83c3-561898a44662-kube-api-access-4d8dw" (OuterVolumeSpecName: "kube-api-access-4d8dw") pod "09afc480-bdfd-4b90-83c3-561898a44662" (UID: "09afc480-bdfd-4b90-83c3-561898a44662"). InnerVolumeSpecName "kube-api-access-4d8dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:31:41.632967 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.632940 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09afc480-bdfd-4b90-83c3-561898a44662-console-oauth-config\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:41.632967 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.632964 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09afc480-bdfd-4b90-83c3-561898a44662-oauth-serving-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:41.632967 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.632973 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d8dw\" (UniqueName: \"kubernetes.io/projected/09afc480-bdfd-4b90-83c3-561898a44662-kube-api-access-4d8dw\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:41.633155 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:41.632982 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09afc480-bdfd-4b90-83c3-561898a44662-console-serving-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:42.088666 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:42.088633 2581 generic.go:356] "Generic (PLEG): container finished" podID="09afc480-bdfd-4b90-83c3-561898a44662" containerID="8bb8b312b5dff25442cd884830da34612220688eafa7ad3fad3dfd9baa355dfd" exitCode=0 Sep 17 15:31:42.088812 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:42.088690 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69dbdbd59-47qx5" Sep 17 15:31:42.088812 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:42.088721 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69dbdbd59-47qx5" event={"ID":"09afc480-bdfd-4b90-83c3-561898a44662","Type":"ContainerDied","Data":"8bb8b312b5dff25442cd884830da34612220688eafa7ad3fad3dfd9baa355dfd"} Sep 17 15:31:42.088812 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:42.088805 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69dbdbd59-47qx5" event={"ID":"09afc480-bdfd-4b90-83c3-561898a44662","Type":"ContainerDied","Data":"df9931c6ca89e5e7c7330f282b1c12739dc372cb7553ab50ce01f00543ee1992"} Sep 17 15:31:42.088939 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:42.088830 2581 scope.go:117] "RemoveContainer" containerID="8bb8b312b5dff25442cd884830da34612220688eafa7ad3fad3dfd9baa355dfd" Sep 17 15:31:42.097437 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:42.097416 2581 scope.go:117] "RemoveContainer" containerID="8bb8b312b5dff25442cd884830da34612220688eafa7ad3fad3dfd9baa355dfd" Sep 17 15:31:42.097702 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:31:42.097683 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb8b312b5dff25442cd884830da34612220688eafa7ad3fad3dfd9baa355dfd\": container with ID starting with 8bb8b312b5dff25442cd884830da34612220688eafa7ad3fad3dfd9baa355dfd not found: ID does not exist" containerID="8bb8b312b5dff25442cd884830da34612220688eafa7ad3fad3dfd9baa355dfd" Sep 17 15:31:42.097752 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:42.097710 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb8b312b5dff25442cd884830da34612220688eafa7ad3fad3dfd9baa355dfd"} err="failed to get container status \"8bb8b312b5dff25442cd884830da34612220688eafa7ad3fad3dfd9baa355dfd\": rpc error: code = NotFound desc = could not find container \"8bb8b312b5dff25442cd884830da34612220688eafa7ad3fad3dfd9baa355dfd\": container with ID starting with 8bb8b312b5dff25442cd884830da34612220688eafa7ad3fad3dfd9baa355dfd not found: ID does not exist" Sep 17 15:31:42.108557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:42.108534 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69dbdbd59-47qx5"] Sep 17 15:31:42.112845 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:42.112824 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69dbdbd59-47qx5"] Sep 17 15:31:43.460606 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:43.460574 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09afc480-bdfd-4b90-83c3-561898a44662" path="/var/lib/kubelet/pods/09afc480-bdfd-4b90-83c3-561898a44662/volumes" Sep 17 15:31:45.252032 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.251996 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bb78f6c45-5x666" podUID="ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d" containerName="console" containerID="cri-o://47c98b014b42e65a80fc9e07eab66901d53ddce7dd005424afeca9dc1a558592" gracePeriod=15 Sep 17 15:31:45.501006 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.500982 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:45.662670 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.662638 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-oauth-serving-cert\") pod \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " Sep 17 15:31:45.662670 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.662668 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6vlv\" (UniqueName: \"kubernetes.io/projected/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-kube-api-access-j6vlv\") pod \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " Sep 17 15:31:45.662895 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.662729 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-trusted-ca-bundle\") pod \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " Sep 17 15:31:45.662895 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.662838 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-serving-cert\") pod \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " Sep 17 15:31:45.662895 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.662876 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-config\") pod \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " Sep 17 15:31:45.663064 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.662905 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-service-ca\") pod \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " Sep 17 15:31:45.663064 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.662971 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-oauth-config\") pod \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\" (UID: \"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d\") " Sep 17 15:31:45.663168 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.663088 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d" (UID: "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:31:45.663168 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.663116 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d" (UID: "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:31:45.663330 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.663306 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-config" (OuterVolumeSpecName: "console-config") pod "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d" (UID: "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:31:45.663330 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.663321 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-trusted-ca-bundle\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:45.663416 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.663322 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-service-ca" (OuterVolumeSpecName: "service-ca") pod "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d" (UID: "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:31:45.663416 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.663340 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-oauth-serving-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:45.665022 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.664996 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-kube-api-access-j6vlv" (OuterVolumeSpecName: "kube-api-access-j6vlv") pod "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d" (UID: "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d"). InnerVolumeSpecName "kube-api-access-j6vlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:31:45.665022 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.665012 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d" (UID: "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:31:45.665156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.665080 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d" (UID: "ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:31:45.763953 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.763900 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-oauth-config\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:45.763953 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.763949 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6vlv\" (UniqueName: \"kubernetes.io/projected/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-kube-api-access-j6vlv\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:45.763953 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.763958 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-serving-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:45.763953 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.763967 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-console-config\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:45.764177 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:45.763976 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:31:46.105183 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:46.105156 2581 generic.go:356] "Generic (PLEG): container finished" podID="ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d" containerID="47c98b014b42e65a80fc9e07eab66901d53ddce7dd005424afeca9dc1a558592" exitCode=0 Sep 17 15:31:46.105333 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:46.105197 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb78f6c45-5x666" Sep 17 15:31:46.105333 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:46.105195 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb78f6c45-5x666" event={"ID":"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d","Type":"ContainerDied","Data":"47c98b014b42e65a80fc9e07eab66901d53ddce7dd005424afeca9dc1a558592"} Sep 17 15:31:46.105333 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:46.105327 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb78f6c45-5x666" event={"ID":"ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d","Type":"ContainerDied","Data":"7f8e059ca7d5ff17d9cf08c9ed7d2faa46045565073e2c33b94584b70955ed87"} Sep 17 15:31:46.105453 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:46.105354 2581 scope.go:117] "RemoveContainer" containerID="47c98b014b42e65a80fc9e07eab66901d53ddce7dd005424afeca9dc1a558592" Sep 17 15:31:46.116842 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:46.116825 2581 scope.go:117] "RemoveContainer" containerID="47c98b014b42e65a80fc9e07eab66901d53ddce7dd005424afeca9dc1a558592" Sep 17 15:31:46.117121 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:31:46.117103 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c98b014b42e65a80fc9e07eab66901d53ddce7dd005424afeca9dc1a558592\": container with ID starting with 47c98b014b42e65a80fc9e07eab66901d53ddce7dd005424afeca9dc1a558592 not found: ID does not exist" containerID="47c98b014b42e65a80fc9e07eab66901d53ddce7dd005424afeca9dc1a558592" Sep 17 15:31:46.117173 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:46.117128 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c98b014b42e65a80fc9e07eab66901d53ddce7dd005424afeca9dc1a558592"} err="failed to get container status \"47c98b014b42e65a80fc9e07eab66901d53ddce7dd005424afeca9dc1a558592\": rpc error: code = NotFound desc = could not find container \"47c98b014b42e65a80fc9e07eab66901d53ddce7dd005424afeca9dc1a558592\": container with ID starting with 47c98b014b42e65a80fc9e07eab66901d53ddce7dd005424afeca9dc1a558592 not found: ID does not exist" Sep 17 15:31:46.135015 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:46.134994 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb78f6c45-5x666"] Sep 17 15:31:46.139826 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:46.139805 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bb78f6c45-5x666"] Sep 17 15:31:47.460539 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:47.460506 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d" path="/var/lib/kubelet/pods/ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d/volumes" Sep 17 15:31:56.023321 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:56.023285 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6mlp5" Sep 17 15:31:58.064103 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:58.064070 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:31:58.196105 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:31:58.196076 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:32:03.272512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:03.272483 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:32:03.461973 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:03.461940 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:32:25.861802 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:25.861770 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:32:25.943671 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:25.943639 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:32:37.888561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:37.888506 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:32:37.978998 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:37.978967 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:32:42.024389 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.024356 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-594df7bf64-p25tc"] Sep 17 15:32:42.024877 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.024744 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d" containerName="console" Sep 17 15:32:42.024877 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.024762 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d" containerName="console" Sep 17 15:32:42.024877 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.024783 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09afc480-bdfd-4b90-83c3-561898a44662" containerName="console" Sep 17 15:32:42.024877 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.024790 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="09afc480-bdfd-4b90-83c3-561898a44662" containerName="console" Sep 17 15:32:42.024877 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.024855 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff5bd915-0ea2-446f-b5e8-80be6a3e1a9d" containerName="console" Sep 17 15:32:42.024877 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.024866 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="09afc480-bdfd-4b90-83c3-561898a44662" containerName="console" Sep 17 15:32:42.026818 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.026797 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.044796 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.044769 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-594df7bf64-p25tc"] Sep 17 15:32:42.059703 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.059682 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-oauth-serving-cert\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.059823 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.059715 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpnkx\" (UniqueName: \"kubernetes.io/projected/e191da55-d622-4b30-801d-c537e6b18270-kube-api-access-bpnkx\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.059823 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.059735 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e191da55-d622-4b30-801d-c537e6b18270-console-oauth-config\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.059823 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.059798 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e191da55-d622-4b30-801d-c537e6b18270-console-serving-cert\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.060006 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.059873 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-service-ca\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.060006 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.059902 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-console-config\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.060006 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.059967 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-trusted-ca-bundle\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.102590 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.102555 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:32:42.160774 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.160750 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-service-ca\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.160858 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.160776 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-console-config\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.160858 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.160793 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-trusted-ca-bundle\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.160858 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.160810 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-oauth-serving-cert\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.160858 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.160849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpnkx\" (UniqueName: \"kubernetes.io/projected/e191da55-d622-4b30-801d-c537e6b18270-kube-api-access-bpnkx\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.161062 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.161043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e191da55-d622-4b30-801d-c537e6b18270-console-oauth-config\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.161115 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.161101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e191da55-d622-4b30-801d-c537e6b18270-console-serving-cert\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.161596 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.161572 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-console-config\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.161596 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.161592 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-service-ca\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.161748 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.161689 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-oauth-serving-cert\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.161748 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.161727 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-trusted-ca-bundle\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.163569 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.163545 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e191da55-d622-4b30-801d-c537e6b18270-console-oauth-config\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.163645 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.163621 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e191da55-d622-4b30-801d-c537e6b18270-console-serving-cert\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.166691 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.166669 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:32:42.169531 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.169508 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpnkx\" (UniqueName: \"kubernetes.io/projected/e191da55-d622-4b30-801d-c537e6b18270-kube-api-access-bpnkx\") pod \"console-594df7bf64-p25tc\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.335235 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.335192 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:42.616350 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:42.616274 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-594df7bf64-p25tc"] Sep 17 15:32:42.618535 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:32:42.618509 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode191da55_d622_4b30_801d_c537e6b18270.slice/crio-a56f4a46db0cbc97a8b9704577052812c344a6387e96ffbfb0592bacaca5fe7c WatchSource:0}: Error finding container a56f4a46db0cbc97a8b9704577052812c344a6387e96ffbfb0592bacaca5fe7c: Status 404 returned error can't find the container with id a56f4a46db0cbc97a8b9704577052812c344a6387e96ffbfb0592bacaca5fe7c Sep 17 15:32:43.309902 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:43.309868 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-594df7bf64-p25tc" event={"ID":"e191da55-d622-4b30-801d-c537e6b18270","Type":"ContainerStarted","Data":"4af340571cfae933af9728bfd1e2aa7fc8e5e6c3c2b5fd5237d130ad3ac68560"} Sep 17 15:32:43.310354 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:43.310002 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-594df7bf64-p25tc" event={"ID":"e191da55-d622-4b30-801d-c537e6b18270","Type":"ContainerStarted","Data":"a56f4a46db0cbc97a8b9704577052812c344a6387e96ffbfb0592bacaca5fe7c"} Sep 17 15:32:43.337379 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:43.337333 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-594df7bf64-p25tc" podStartSLOduration=1.3373147429999999 podStartE2EDuration="1.337314743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:32:43.336586865 +0000 UTC m=+150.332823612" watchObservedRunningTime="2026-09-17 15:32:43.337314743 +0000 UTC m=+150.333551489" Sep 17 15:32:48.008572 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:48.008536 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:32:48.126762 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:48.126729 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:32:52.335406 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:52.335374 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:52.335805 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:52.335441 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:52.340235 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:52.340211 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:52.344545 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:52.344525 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:32:52.483957 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:52.483906 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5db856bc-x8hgh"] Sep 17 15:32:52.717869 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:52.717789 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:32:52.837851 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:32:52.837815 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:04.852482 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:04.852442 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:13.921895 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:13.921814 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:14.013307 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:14.013273 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:15.969542 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:15.969508 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8"] Sep 17 15:33:15.972130 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:15.972109 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" Sep 17 15:33:15.975976 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:15.975953 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 15:33:15.976125 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:15.976104 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5lr7h\"" Sep 17 15:33:15.976192 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:15.976137 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 15:33:15.995815 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:15.995793 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8"] Sep 17 15:33:16.026636 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:16.026613 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:16.093126 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:16.093103 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:16.111581 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:16.111560 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba2b4439-19bb-4941-b139-d137070fc670-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8\" (UID: \"ba2b4439-19bb-4941-b139-d137070fc670\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" Sep 17 15:33:16.111643 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:16.111592 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2r6j\" (UniqueName: \"kubernetes.io/projected/ba2b4439-19bb-4941-b139-d137070fc670-kube-api-access-w2r6j\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8\" (UID: \"ba2b4439-19bb-4941-b139-d137070fc670\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" Sep 17 15:33:16.111643 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:16.111615 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba2b4439-19bb-4941-b139-d137070fc670-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8\" (UID: \"ba2b4439-19bb-4941-b139-d137070fc670\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" Sep 17 15:33:16.212298 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:16.212256 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2r6j\" (UniqueName: \"kubernetes.io/projected/ba2b4439-19bb-4941-b139-d137070fc670-kube-api-access-w2r6j\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8\" (UID: \"ba2b4439-19bb-4941-b139-d137070fc670\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" Sep 17 15:33:16.212412 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:16.212314 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba2b4439-19bb-4941-b139-d137070fc670-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8\" (UID: \"ba2b4439-19bb-4941-b139-d137070fc670\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" Sep 17 15:33:16.212412 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:16.212395 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba2b4439-19bb-4941-b139-d137070fc670-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8\" (UID: \"ba2b4439-19bb-4941-b139-d137070fc670\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" Sep 17 15:33:16.212686 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:16.212668 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba2b4439-19bb-4941-b139-d137070fc670-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8\" (UID: \"ba2b4439-19bb-4941-b139-d137070fc670\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" Sep 17 15:33:16.212722 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:16.212699 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba2b4439-19bb-4941-b139-d137070fc670-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8\" (UID: \"ba2b4439-19bb-4941-b139-d137070fc670\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" Sep 17 15:33:16.220610 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:16.220549 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2r6j\" (UniqueName: \"kubernetes.io/projected/ba2b4439-19bb-4941-b139-d137070fc670-kube-api-access-w2r6j\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8\" (UID: \"ba2b4439-19bb-4941-b139-d137070fc670\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" Sep 17 15:33:16.281092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:16.281073 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" Sep 17 15:33:16.550489 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:16.550465 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8"] Sep 17 15:33:16.552815 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:33:16.552783 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2b4439_19bb_4941_b139_d137070fc670.slice/crio-a35ead93626593a4687e743682fdbadd25d8f87af5f96f3da5bb0ed59db87028 WatchSource:0}: Error finding container a35ead93626593a4687e743682fdbadd25d8f87af5f96f3da5bb0ed59db87028: Status 404 returned error can't find the container with id a35ead93626593a4687e743682fdbadd25d8f87af5f96f3da5bb0ed59db87028 Sep 17 15:33:17.428303 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.428250 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" event={"ID":"ba2b4439-19bb-4941-b139-d137070fc670","Type":"ContainerStarted","Data":"a35ead93626593a4687e743682fdbadd25d8f87af5f96f3da5bb0ed59db87028"} Sep 17 15:33:17.502512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.502473 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f5db856bc-x8hgh" podUID="2bdda5b9-6e31-4e66-9ec7-86f1920f82a6" containerName="console" containerID="cri-o://32379253f9fff78139eae5ecb989c4e89160415fea16350931218f2f08c87520" gracePeriod=15 Sep 17 15:33:17.732665 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.732641 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:33:17.825283 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.825252 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-oauth-serving-cert\") pod \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " Sep 17 15:33:17.825283 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.825315 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7glxn\" (UniqueName: \"kubernetes.io/projected/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-kube-api-access-7glxn\") pod \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " Sep 17 15:33:17.825632 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.825345 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-service-ca\") pod \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " Sep 17 15:33:17.825632 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.825378 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-oauth-config\") pod \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " Sep 17 15:33:17.825632 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.825405 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-config\") pod \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " Sep 17 15:33:17.825632 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.825437 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-trusted-ca-bundle\") pod \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " Sep 17 15:33:17.825632 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.825460 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-serving-cert\") pod \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\" (UID: \"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6\") " Sep 17 15:33:17.825940 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.825708 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6" (UID: "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:33:17.825940 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.825889 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-service-ca" (OuterVolumeSpecName: "service-ca") pod "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6" (UID: "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:33:17.826074 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.826037 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6" (UID: "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:33:17.826152 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.826126 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-config" (OuterVolumeSpecName: "console-config") pod "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6" (UID: "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:33:17.828178 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.828150 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-kube-api-access-7glxn" (OuterVolumeSpecName: "kube-api-access-7glxn") pod "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6" (UID: "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6"). InnerVolumeSpecName "kube-api-access-7glxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:33:17.828291 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.828189 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6" (UID: "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:33:17.828291 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.828254 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6" (UID: "2bdda5b9-6e31-4e66-9ec7-86f1920f82a6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:33:17.926048 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.926022 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-oauth-serving-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:33:17.926048 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.926044 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7glxn\" (UniqueName: \"kubernetes.io/projected/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-kube-api-access-7glxn\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:33:17.926182 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.926055 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:33:17.926182 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.926063 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-oauth-config\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:33:17.926182 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.926072 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-config\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:33:17.926182 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.926080 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-trusted-ca-bundle\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:33:17.926182 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:17.926088 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6-console-serving-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:33:18.433521 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:18.433483 2581 generic.go:356] "Generic (PLEG): container finished" podID="2bdda5b9-6e31-4e66-9ec7-86f1920f82a6" containerID="32379253f9fff78139eae5ecb989c4e89160415fea16350931218f2f08c87520" exitCode=0 Sep 17 15:33:18.434001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:18.433534 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5db856bc-x8hgh" Sep 17 15:33:18.434001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:18.433582 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5db856bc-x8hgh" event={"ID":"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6","Type":"ContainerDied","Data":"32379253f9fff78139eae5ecb989c4e89160415fea16350931218f2f08c87520"} Sep 17 15:33:18.434001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:18.433646 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5db856bc-x8hgh" event={"ID":"2bdda5b9-6e31-4e66-9ec7-86f1920f82a6","Type":"ContainerDied","Data":"84d2afbf6645113a9d33621d86d273862ac21bd233f517856b51d59f666aaa7b"} Sep 17 15:33:18.434001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:18.433677 2581 scope.go:117] "RemoveContainer" containerID="32379253f9fff78139eae5ecb989c4e89160415fea16350931218f2f08c87520" Sep 17 15:33:18.443490 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:18.443469 2581 scope.go:117] "RemoveContainer" containerID="32379253f9fff78139eae5ecb989c4e89160415fea16350931218f2f08c87520" Sep 17 15:33:18.443777 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:33:18.443749 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32379253f9fff78139eae5ecb989c4e89160415fea16350931218f2f08c87520\": container with ID starting with 32379253f9fff78139eae5ecb989c4e89160415fea16350931218f2f08c87520 not found: ID does not exist" containerID="32379253f9fff78139eae5ecb989c4e89160415fea16350931218f2f08c87520" Sep 17 15:33:18.443856 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:18.443788 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32379253f9fff78139eae5ecb989c4e89160415fea16350931218f2f08c87520"} err="failed to get container status \"32379253f9fff78139eae5ecb989c4e89160415fea16350931218f2f08c87520\": rpc error: code = NotFound desc = could not find container \"32379253f9fff78139eae5ecb989c4e89160415fea16350931218f2f08c87520\": container with ID starting with 32379253f9fff78139eae5ecb989c4e89160415fea16350931218f2f08c87520 not found: ID does not exist" Sep 17 15:33:18.454698 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:18.454678 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5db856bc-x8hgh"] Sep 17 15:33:18.458802 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:18.458781 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f5db856bc-x8hgh"] Sep 17 15:33:19.459519 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:19.459490 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bdda5b9-6e31-4e66-9ec7-86f1920f82a6" path="/var/lib/kubelet/pods/2bdda5b9-6e31-4e66-9ec7-86f1920f82a6/volumes" Sep 17 15:33:23.453631 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:23.453591 2581 generic.go:356] "Generic (PLEG): container finished" podID="ba2b4439-19bb-4941-b139-d137070fc670" containerID="232168dd80e6b732795439feac8c09750eb2640e32297dcc4d99afd07c35111e" exitCode=0 Sep 17 15:33:23.454106 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:23.453662 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" event={"ID":"ba2b4439-19bb-4941-b139-d137070fc670","Type":"ContainerDied","Data":"232168dd80e6b732795439feac8c09750eb2640e32297dcc4d99afd07c35111e"} Sep 17 15:33:26.468534 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:26.468500 2581 generic.go:356] "Generic (PLEG): container finished" podID="ba2b4439-19bb-4941-b139-d137070fc670" containerID="15c2859a4979830efdca7e56868fd333f1793b2bba7d376f9d13fbccf4e5cd49" exitCode=0 Sep 17 15:33:26.468993 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:26.468560 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" event={"ID":"ba2b4439-19bb-4941-b139-d137070fc670","Type":"ContainerDied","Data":"15c2859a4979830efdca7e56868fd333f1793b2bba7d376f9d13fbccf4e5cd49"} Sep 17 15:33:33.251147 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:33.251101 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:33.316561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:33.316532 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:33.496074 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:33.496042 2581 generic.go:356] "Generic (PLEG): container finished" podID="ba2b4439-19bb-4941-b139-d137070fc670" containerID="fa9a416c345bd4c16c4e47c964f37e1c4802f0182fbf7364302fbb3e63ee5800" exitCode=0 Sep 17 15:33:33.496241 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:33.496109 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" event={"ID":"ba2b4439-19bb-4941-b139-d137070fc670","Type":"ContainerDied","Data":"fa9a416c345bd4c16c4e47c964f37e1c4802f0182fbf7364302fbb3e63ee5800"} Sep 17 15:33:34.623055 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:34.623033 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" Sep 17 15:33:34.658189 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:34.658157 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba2b4439-19bb-4941-b139-d137070fc670-bundle\") pod \"ba2b4439-19bb-4941-b139-d137070fc670\" (UID: \"ba2b4439-19bb-4941-b139-d137070fc670\") " Sep 17 15:33:34.658348 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:34.658203 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba2b4439-19bb-4941-b139-d137070fc670-util\") pod \"ba2b4439-19bb-4941-b139-d137070fc670\" (UID: \"ba2b4439-19bb-4941-b139-d137070fc670\") " Sep 17 15:33:34.658348 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:34.658280 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2r6j\" (UniqueName: \"kubernetes.io/projected/ba2b4439-19bb-4941-b139-d137070fc670-kube-api-access-w2r6j\") pod \"ba2b4439-19bb-4941-b139-d137070fc670\" (UID: \"ba2b4439-19bb-4941-b139-d137070fc670\") " Sep 17 15:33:34.659101 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:34.659070 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2b4439-19bb-4941-b139-d137070fc670-bundle" (OuterVolumeSpecName: "bundle") pod "ba2b4439-19bb-4941-b139-d137070fc670" (UID: "ba2b4439-19bb-4941-b139-d137070fc670"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:33:34.660814 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:34.660779 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba2b4439-19bb-4941-b139-d137070fc670-kube-api-access-w2r6j" (OuterVolumeSpecName: "kube-api-access-w2r6j") pod "ba2b4439-19bb-4941-b139-d137070fc670" (UID: "ba2b4439-19bb-4941-b139-d137070fc670"). InnerVolumeSpecName "kube-api-access-w2r6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:33:34.663069 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:34.663042 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2b4439-19bb-4941-b139-d137070fc670-util" (OuterVolumeSpecName: "util") pod "ba2b4439-19bb-4941-b139-d137070fc670" (UID: "ba2b4439-19bb-4941-b139-d137070fc670"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:33:34.759476 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:34.759409 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2r6j\" (UniqueName: \"kubernetes.io/projected/ba2b4439-19bb-4941-b139-d137070fc670-kube-api-access-w2r6j\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:33:34.759476 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:34.759445 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba2b4439-19bb-4941-b139-d137070fc670-bundle\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:33:34.759476 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:34.759455 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba2b4439-19bb-4941-b139-d137070fc670-util\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:33:35.504425 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:35.504389 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" event={"ID":"ba2b4439-19bb-4941-b139-d137070fc670","Type":"ContainerDied","Data":"a35ead93626593a4687e743682fdbadd25d8f87af5f96f3da5bb0ed59db87028"} Sep 17 15:33:35.504425 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:35.504411 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409hvl8" Sep 17 15:33:35.504576 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:35.504430 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a35ead93626593a4687e743682fdbadd25d8f87af5f96f3da5bb0ed59db87028" Sep 17 15:33:37.207678 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:37.207641 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:37.518219 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:37.518151 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:39.371192 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.371158 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5"] Sep 17 15:33:39.371529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.371417 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba2b4439-19bb-4941-b139-d137070fc670" containerName="pull" Sep 17 15:33:39.371529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.371429 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba2b4439-19bb-4941-b139-d137070fc670" containerName="pull" Sep 17 15:33:39.371529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.371443 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba2b4439-19bb-4941-b139-d137070fc670" containerName="extract" Sep 17 15:33:39.371529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.371450 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba2b4439-19bb-4941-b139-d137070fc670" containerName="extract" Sep 17 15:33:39.371529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.371464 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba2b4439-19bb-4941-b139-d137070fc670" containerName="util" Sep 17 15:33:39.371529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.371471 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba2b4439-19bb-4941-b139-d137070fc670" containerName="util" Sep 17 15:33:39.371529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.371483 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bdda5b9-6e31-4e66-9ec7-86f1920f82a6" containerName="console" Sep 17 15:33:39.371529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.371488 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bdda5b9-6e31-4e66-9ec7-86f1920f82a6" containerName="console" Sep 17 15:33:39.371764 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.371533 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bdda5b9-6e31-4e66-9ec7-86f1920f82a6" containerName="console" Sep 17 15:33:39.371764 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.371542 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba2b4439-19bb-4941-b139-d137070fc670" containerName="extract" Sep 17 15:33:39.400761 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.400734 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5"] Sep 17 15:33:39.400958 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.400879 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5" Sep 17 15:33:39.404580 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.404554 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 17 15:33:39.404704 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.404612 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jchbs\"" Sep 17 15:33:39.404704 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.404628 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 17 15:33:39.493981 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.493949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd25f\" (UniqueName: \"kubernetes.io/projected/0a6bb3b2-64ce-40e7-a535-51ec1395fc2c-kube-api-access-hd25f\") pod \"cert-manager-operator-controller-manager-55c444bf86-tlkd5\" (UID: \"0a6bb3b2-64ce-40e7-a535-51ec1395fc2c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5" Sep 17 15:33:39.494122 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.494025 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a6bb3b2-64ce-40e7-a535-51ec1395fc2c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tlkd5\" (UID: \"0a6bb3b2-64ce-40e7-a535-51ec1395fc2c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5" Sep 17 15:33:39.594788 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.594759 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a6bb3b2-64ce-40e7-a535-51ec1395fc2c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tlkd5\" (UID: \"0a6bb3b2-64ce-40e7-a535-51ec1395fc2c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5" Sep 17 15:33:39.594944 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.594801 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd25f\" (UniqueName: \"kubernetes.io/projected/0a6bb3b2-64ce-40e7-a535-51ec1395fc2c-kube-api-access-hd25f\") pod \"cert-manager-operator-controller-manager-55c444bf86-tlkd5\" (UID: \"0a6bb3b2-64ce-40e7-a535-51ec1395fc2c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5" Sep 17 15:33:39.595220 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.595199 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a6bb3b2-64ce-40e7-a535-51ec1395fc2c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tlkd5\" (UID: \"0a6bb3b2-64ce-40e7-a535-51ec1395fc2c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5" Sep 17 15:33:39.602701 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.602676 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd25f\" (UniqueName: \"kubernetes.io/projected/0a6bb3b2-64ce-40e7-a535-51ec1395fc2c-kube-api-access-hd25f\") pod \"cert-manager-operator-controller-manager-55c444bf86-tlkd5\" (UID: \"0a6bb3b2-64ce-40e7-a535-51ec1395fc2c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5" Sep 17 15:33:39.709679 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:39.709616 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5" Sep 17 15:33:40.010765 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:40.010741 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5"] Sep 17 15:33:40.013607 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:33:40.013582 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6bb3b2_64ce_40e7_a535_51ec1395fc2c.slice/crio-3f8f61b75b74f01a44f42db12f56f7ab7cc5dd87c760b3b80371fa883d9a6e8f WatchSource:0}: Error finding container 3f8f61b75b74f01a44f42db12f56f7ab7cc5dd87c760b3b80371fa883d9a6e8f: Status 404 returned error can't find the container with id 3f8f61b75b74f01a44f42db12f56f7ab7cc5dd87c760b3b80371fa883d9a6e8f Sep 17 15:33:40.523563 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:40.523530 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5" event={"ID":"0a6bb3b2-64ce-40e7-a535-51ec1395fc2c","Type":"ContainerStarted","Data":"3f8f61b75b74f01a44f42db12f56f7ab7cc5dd87c760b3b80371fa883d9a6e8f"} Sep 17 15:33:40.989171 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:40.989137 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:41.090289 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:41.090254 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:45.863664 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:45.863630 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:46.070015 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:46.069981 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:48.560322 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:48.560272 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5" event={"ID":"0a6bb3b2-64ce-40e7-a535-51ec1395fc2c","Type":"ContainerStarted","Data":"ce15ade439fdc3b245cfbaa976120cbe8540a0af908924e04b3a92e1909b49d5"} Sep 17 15:33:48.856346 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:48.856270 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:48.960084 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:48.960052 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:49.101303 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:49.101265 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:49.206290 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:49.206212 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:49.864162 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:49.864110 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tlkd5" podStartSLOduration=2.8776501469999998 podStartE2EDuration="10.864091105s" totalImagesPullingTime="7.986440958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:33:48.586145319 +0000 UTC m=+215.582382068" watchObservedRunningTime="2026-09-17 15:33:49.864091105 +0000 UTC m=+216.860327850" Sep 17 15:33:49.864537 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:49.864495 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6d9tr"] Sep 17 15:33:49.866461 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:49.866446 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9tr" Sep 17 15:33:49.869600 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:49.869580 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ft8xx\"" Sep 17 15:33:49.870338 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:49.870319 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 17 15:33:49.870412 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:49.870364 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 17 15:33:49.881238 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:49.881221 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6d9tr"] Sep 17 15:33:49.967878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:49.967853 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52c86\" (UniqueName: \"kubernetes.io/projected/b8bc3221-c24d-471d-8458-0a596e6fb5b6-kube-api-access-52c86\") pod \"cert-manager-webhook-6486ffd997-6d9tr\" (UID: \"b8bc3221-c24d-471d-8458-0a596e6fb5b6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6d9tr" Sep 17 15:33:49.968017 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:49.967909 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8bc3221-c24d-471d-8458-0a596e6fb5b6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6d9tr\" (UID: \"b8bc3221-c24d-471d-8458-0a596e6fb5b6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6d9tr" Sep 17 15:33:50.069113 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.069080 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52c86\" (UniqueName: \"kubernetes.io/projected/b8bc3221-c24d-471d-8458-0a596e6fb5b6-kube-api-access-52c86\") pod \"cert-manager-webhook-6486ffd997-6d9tr\" (UID: \"b8bc3221-c24d-471d-8458-0a596e6fb5b6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6d9tr" Sep 17 15:33:50.069245 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.069161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8bc3221-c24d-471d-8458-0a596e6fb5b6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6d9tr\" (UID: \"b8bc3221-c24d-471d-8458-0a596e6fb5b6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6d9tr" Sep 17 15:33:50.077207 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.077184 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8bc3221-c24d-471d-8458-0a596e6fb5b6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6d9tr\" (UID: \"b8bc3221-c24d-471d-8458-0a596e6fb5b6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6d9tr" Sep 17 15:33:50.077614 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.077596 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52c86\" (UniqueName: \"kubernetes.io/projected/b8bc3221-c24d-471d-8458-0a596e6fb5b6-kube-api-access-52c86\") pod \"cert-manager-webhook-6486ffd997-6d9tr\" (UID: \"b8bc3221-c24d-471d-8458-0a596e6fb5b6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6d9tr" Sep 17 15:33:50.184745 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.184678 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9tr" Sep 17 15:33:50.455634 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.455609 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6d9tr"] Sep 17 15:33:50.458133 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:33:50.458105 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8bc3221_c24d_471d_8458_0a596e6fb5b6.slice/crio-f6bdd554ad189f19b246d098abcc754b6134851520e1371d8e791162dff01009 WatchSource:0}: Error finding container f6bdd554ad189f19b246d098abcc754b6134851520e1371d8e791162dff01009: Status 404 returned error can't find the container with id f6bdd554ad189f19b246d098abcc754b6134851520e1371d8e791162dff01009 Sep 17 15:33:50.569662 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.569634 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9tr" event={"ID":"b8bc3221-c24d-471d-8458-0a596e6fb5b6","Type":"ContainerStarted","Data":"f6bdd554ad189f19b246d098abcc754b6134851520e1371d8e791162dff01009"} Sep 17 15:33:50.668995 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.668962 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t"] Sep 17 15:33:50.671996 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.671980 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" Sep 17 15:33:50.674547 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.674523 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 15:33:50.674662 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.674559 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5lr7h\"" Sep 17 15:33:50.674662 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.674588 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 15:33:50.683506 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.683484 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t"] Sep 17 15:33:50.774771 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.774701 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/806bbd59-7f93-408d-8511-c8595bcf9553-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t\" (UID: \"806bbd59-7f93-408d-8511-c8595bcf9553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" Sep 17 15:33:50.774771 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.774759 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pgd2\" (UniqueName: \"kubernetes.io/projected/806bbd59-7f93-408d-8511-c8595bcf9553-kube-api-access-9pgd2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t\" (UID: \"806bbd59-7f93-408d-8511-c8595bcf9553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" Sep 17 15:33:50.774942 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.774796 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/806bbd59-7f93-408d-8511-c8595bcf9553-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t\" (UID: \"806bbd59-7f93-408d-8511-c8595bcf9553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" Sep 17 15:33:50.875683 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.875650 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/806bbd59-7f93-408d-8511-c8595bcf9553-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t\" (UID: \"806bbd59-7f93-408d-8511-c8595bcf9553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" Sep 17 15:33:50.876026 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.875734 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/806bbd59-7f93-408d-8511-c8595bcf9553-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t\" (UID: \"806bbd59-7f93-408d-8511-c8595bcf9553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" Sep 17 15:33:50.876026 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.875755 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pgd2\" (UniqueName: \"kubernetes.io/projected/806bbd59-7f93-408d-8511-c8595bcf9553-kube-api-access-9pgd2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t\" (UID: \"806bbd59-7f93-408d-8511-c8595bcf9553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" Sep 17 15:33:50.876026 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.876013 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/806bbd59-7f93-408d-8511-c8595bcf9553-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t\" (UID: \"806bbd59-7f93-408d-8511-c8595bcf9553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" Sep 17 15:33:50.876130 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.876105 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/806bbd59-7f93-408d-8511-c8595bcf9553-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t\" (UID: \"806bbd59-7f93-408d-8511-c8595bcf9553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" Sep 17 15:33:50.883593 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.883567 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pgd2\" (UniqueName: \"kubernetes.io/projected/806bbd59-7f93-408d-8511-c8595bcf9553-kube-api-access-9pgd2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t\" (UID: \"806bbd59-7f93-408d-8511-c8595bcf9553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" Sep 17 15:33:50.981277 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:50.981252 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" Sep 17 15:33:51.266878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.266839 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t"] Sep 17 15:33:51.268119 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:33:51.268087 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806bbd59_7f93_408d_8511_c8595bcf9553.slice/crio-405d61b171ae1fd141366712504c31afff6aab01575bff870e574908f5b1570f WatchSource:0}: Error finding container 405d61b171ae1fd141366712504c31afff6aab01575bff870e574908f5b1570f: Status 404 returned error can't find the container with id 405d61b171ae1fd141366712504c31afff6aab01575bff870e574908f5b1570f Sep 17 15:33:51.272037 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.272016 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:51.367884 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.367859 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:51.468565 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.468531 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:33:51.576463 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.576379 2581 generic.go:356] "Generic (PLEG): container finished" podID="806bbd59-7f93-408d-8511-c8595bcf9553" containerID="6429a08f4c74ee710d14c7735773b4b59ddfa1a559cf67febc6d755bcc39bcdb" exitCode=0 Sep 17 15:33:51.576463 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.576457 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" event={"ID":"806bbd59-7f93-408d-8511-c8595bcf9553","Type":"ContainerDied","Data":"6429a08f4c74ee710d14c7735773b4b59ddfa1a559cf67febc6d755bcc39bcdb"} Sep 17 15:33:51.576661 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.576500 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" event={"ID":"806bbd59-7f93-408d-8511-c8595bcf9553","Type":"ContainerStarted","Data":"405d61b171ae1fd141366712504c31afff6aab01575bff870e574908f5b1570f"} Sep 17 15:33:51.693799 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.693761 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj"] Sep 17 15:33:51.696270 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.696248 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj" Sep 17 15:33:51.699307 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.699289 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-k9v5p\"" Sep 17 15:33:51.707661 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.707639 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj"] Sep 17 15:33:51.783543 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.783501 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjrds\" (UniqueName: \"kubernetes.io/projected/2f741599-ff2e-4c5d-aa46-0a659f73adb1-kube-api-access-rjrds\") pod \"cert-manager-cainjector-559cdd8c87-v5zlj\" (UID: \"2f741599-ff2e-4c5d-aa46-0a659f73adb1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj" Sep 17 15:33:51.783739 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.783572 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f741599-ff2e-4c5d-aa46-0a659f73adb1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-v5zlj\" (UID: \"2f741599-ff2e-4c5d-aa46-0a659f73adb1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj" Sep 17 15:33:51.884297 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.884203 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f741599-ff2e-4c5d-aa46-0a659f73adb1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-v5zlj\" (UID: \"2f741599-ff2e-4c5d-aa46-0a659f73adb1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj" Sep 17 15:33:51.884725 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.884304 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrds\" (UniqueName: \"kubernetes.io/projected/2f741599-ff2e-4c5d-aa46-0a659f73adb1-kube-api-access-rjrds\") pod \"cert-manager-cainjector-559cdd8c87-v5zlj\" (UID: \"2f741599-ff2e-4c5d-aa46-0a659f73adb1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj" Sep 17 15:33:51.894902 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.894873 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjrds\" (UniqueName: \"kubernetes.io/projected/2f741599-ff2e-4c5d-aa46-0a659f73adb1-kube-api-access-rjrds\") pod \"cert-manager-cainjector-559cdd8c87-v5zlj\" (UID: \"2f741599-ff2e-4c5d-aa46-0a659f73adb1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj" Sep 17 15:33:51.895157 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:51.895139 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f741599-ff2e-4c5d-aa46-0a659f73adb1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-v5zlj\" (UID: \"2f741599-ff2e-4c5d-aa46-0a659f73adb1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj" Sep 17 15:33:52.006967 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:52.006935 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj" Sep 17 15:33:52.282888 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:52.282630 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj"] Sep 17 15:33:52.285412 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:33:52.285312 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f741599_ff2e_4c5d_aa46_0a659f73adb1.slice/crio-d2c3c58438a793bb6715cf10c8e4aab0b2b969c47e44fdb22a6e293202ad584c WatchSource:0}: Error finding container d2c3c58438a793bb6715cf10c8e4aab0b2b969c47e44fdb22a6e293202ad584c: Status 404 returned error can't find the container with id d2c3c58438a793bb6715cf10c8e4aab0b2b969c47e44fdb22a6e293202ad584c Sep 17 15:33:52.583610 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:52.583512 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj" event={"ID":"2f741599-ff2e-4c5d-aa46-0a659f73adb1","Type":"ContainerStarted","Data":"d2c3c58438a793bb6715cf10c8e4aab0b2b969c47e44fdb22a6e293202ad584c"} Sep 17 15:33:54.593678 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:54.593647 2581 generic.go:356] "Generic (PLEG): container finished" podID="806bbd59-7f93-408d-8511-c8595bcf9553" containerID="0646e29cd1e9d15abb28c0baccf7a81d349b91ff98c1012a12bde0ab59b0a570" exitCode=0 Sep 17 15:33:54.594091 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:54.593732 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" event={"ID":"806bbd59-7f93-408d-8511-c8595bcf9553","Type":"ContainerDied","Data":"0646e29cd1e9d15abb28c0baccf7a81d349b91ff98c1012a12bde0ab59b0a570"} Sep 17 15:33:54.595478 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:54.595456 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj" event={"ID":"2f741599-ff2e-4c5d-aa46-0a659f73adb1","Type":"ContainerStarted","Data":"6f8ec5e0138bd1980ee6ea539229b7992e34b4bea974badd1df3790838e205e0"} Sep 17 15:33:54.597288 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:54.597266 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9tr" event={"ID":"b8bc3221-c24d-471d-8458-0a596e6fb5b6","Type":"ContainerStarted","Data":"1b2863e9cfbe0ed1f4071592c30864aefaac8907d67b95160c2e20cde47b8ba7"} Sep 17 15:33:54.597414 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:54.597367 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9tr" Sep 17 15:33:54.631197 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:54.631136 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9tr" podStartSLOduration=2.008885075 podStartE2EDuration="5.631113695s" totalImagesPullingTime="3.62222862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:33:54.630083517 +0000 UTC m=+221.626320263" watchObservedRunningTime="2026-09-17 15:33:54.631113695 +0000 UTC m=+221.627350442" Sep 17 15:33:54.649626 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:54.649586 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-v5zlj" podStartSLOduration=1.8202310800000001 podStartE2EDuration="3.649569788s" totalImagesPullingTime="1.829338708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:33:54.649078606 +0000 UTC m=+221.645315353" watchObservedRunningTime="2026-09-17 15:33:54.649569788 +0000 UTC m=+221.645806535" Sep 17 15:33:55.603434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:55.603397 2581 generic.go:356] "Generic (PLEG): container finished" podID="806bbd59-7f93-408d-8511-c8595bcf9553" containerID="e33a074d1fe5ae6dcc361a95dbe52fa8ac6528f19cd46455ec47df88c971ca75" exitCode=0 Sep 17 15:33:55.603836 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:55.603485 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" event={"ID":"806bbd59-7f93-408d-8511-c8595bcf9553","Type":"ContainerDied","Data":"e33a074d1fe5ae6dcc361a95dbe52fa8ac6528f19cd46455ec47df88c971ca75"} Sep 17 15:33:56.729000 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:56.728973 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" Sep 17 15:33:56.822931 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:56.822885 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pgd2\" (UniqueName: \"kubernetes.io/projected/806bbd59-7f93-408d-8511-c8595bcf9553-kube-api-access-9pgd2\") pod \"806bbd59-7f93-408d-8511-c8595bcf9553\" (UID: \"806bbd59-7f93-408d-8511-c8595bcf9553\") " Sep 17 15:33:56.823096 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:56.822984 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/806bbd59-7f93-408d-8511-c8595bcf9553-util\") pod \"806bbd59-7f93-408d-8511-c8595bcf9553\" (UID: \"806bbd59-7f93-408d-8511-c8595bcf9553\") " Sep 17 15:33:56.823096 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:56.823025 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/806bbd59-7f93-408d-8511-c8595bcf9553-bundle\") pod \"806bbd59-7f93-408d-8511-c8595bcf9553\" (UID: \"806bbd59-7f93-408d-8511-c8595bcf9553\") " Sep 17 15:33:56.823421 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:56.823389 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806bbd59-7f93-408d-8511-c8595bcf9553-bundle" (OuterVolumeSpecName: "bundle") pod "806bbd59-7f93-408d-8511-c8595bcf9553" (UID: "806bbd59-7f93-408d-8511-c8595bcf9553"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:33:56.824948 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:56.824927 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806bbd59-7f93-408d-8511-c8595bcf9553-kube-api-access-9pgd2" (OuterVolumeSpecName: "kube-api-access-9pgd2") pod "806bbd59-7f93-408d-8511-c8595bcf9553" (UID: "806bbd59-7f93-408d-8511-c8595bcf9553"). InnerVolumeSpecName "kube-api-access-9pgd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:33:56.852335 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:56.852310 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806bbd59-7f93-408d-8511-c8595bcf9553-util" (OuterVolumeSpecName: "util") pod "806bbd59-7f93-408d-8511-c8595bcf9553" (UID: "806bbd59-7f93-408d-8511-c8595bcf9553"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:33:56.923874 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:56.923821 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pgd2\" (UniqueName: \"kubernetes.io/projected/806bbd59-7f93-408d-8511-c8595bcf9553-kube-api-access-9pgd2\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:33:56.923874 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:56.923842 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/806bbd59-7f93-408d-8511-c8595bcf9553-util\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:33:56.923874 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:56.923853 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/806bbd59-7f93-408d-8511-c8595bcf9553-bundle\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:33:57.613432 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:57.613397 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" event={"ID":"806bbd59-7f93-408d-8511-c8595bcf9553","Type":"ContainerDied","Data":"405d61b171ae1fd141366712504c31afff6aab01575bff870e574908f5b1570f"} Sep 17 15:33:57.613622 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:57.613439 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkq7t" Sep 17 15:33:57.613622 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:57.613446 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="405d61b171ae1fd141366712504c31afff6aab01575bff870e574908f5b1570f" Sep 17 15:33:58.651382 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:33:58.651328 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:00.606248 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:00.606217 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6d9tr" Sep 17 15:34:02.373285 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:02.373249 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:02.515107 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:02.515059 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:08.289351 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:08.289316 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:08.393330 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:08.393298 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:10.001028 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.000996 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:10.110067 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.110041 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc"] Sep 17 15:34:10.110307 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.110296 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="806bbd59-7f93-408d-8511-c8595bcf9553" containerName="pull" Sep 17 15:34:10.110349 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.110309 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="806bbd59-7f93-408d-8511-c8595bcf9553" containerName="pull" Sep 17 15:34:10.110349 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.110319 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="806bbd59-7f93-408d-8511-c8595bcf9553" containerName="util" Sep 17 15:34:10.110349 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.110325 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="806bbd59-7f93-408d-8511-c8595bcf9553" containerName="util" Sep 17 15:34:10.110349 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.110342 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="806bbd59-7f93-408d-8511-c8595bcf9553" containerName="extract" Sep 17 15:34:10.110349 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.110348 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="806bbd59-7f93-408d-8511-c8595bcf9553" containerName="extract" Sep 17 15:34:10.110534 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.110396 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="806bbd59-7f93-408d-8511-c8595bcf9553" containerName="extract" Sep 17 15:34:10.112498 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.112481 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" Sep 17 15:34:10.114811 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.114791 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5lr7h\"" Sep 17 15:34:10.114811 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.114805 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 15:34:10.115769 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.115749 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 15:34:10.123867 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.123847 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc"] Sep 17 15:34:10.224298 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.224262 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc\" (UID: \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" Sep 17 15:34:10.224496 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.224316 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc\" (UID: \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" Sep 17 15:34:10.224496 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.224424 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mblr4\" (UniqueName: \"kubernetes.io/projected/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-kube-api-access-mblr4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc\" (UID: \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" Sep 17 15:34:10.324715 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.324680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc\" (UID: \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" Sep 17 15:34:10.324847 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.324743 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mblr4\" (UniqueName: \"kubernetes.io/projected/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-kube-api-access-mblr4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc\" (UID: \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" Sep 17 15:34:10.324847 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.324781 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc\" (UID: \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" Sep 17 15:34:10.325068 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.325049 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc\" (UID: \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" Sep 17 15:34:10.325111 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.325083 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc\" (UID: \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" Sep 17 15:34:10.333562 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.333541 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mblr4\" (UniqueName: \"kubernetes.io/projected/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-kube-api-access-mblr4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc\" (UID: \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" Sep 17 15:34:10.421645 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.421617 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" Sep 17 15:34:10.691203 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:10.691120 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc"] Sep 17 15:34:10.694632 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:34:10.694610 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc39e16b_28fe_413b_bbe0_74418f9a2c0a.slice/crio-ecb7502fd5fe367e99e1eca3450eed57d98a72caa2df1ab453a589dc3c001fb3 WatchSource:0}: Error finding container ecb7502fd5fe367e99e1eca3450eed57d98a72caa2df1ab453a589dc3c001fb3: Status 404 returned error can't find the container with id ecb7502fd5fe367e99e1eca3450eed57d98a72caa2df1ab453a589dc3c001fb3 Sep 17 15:34:11.671664 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:11.671584 2581 generic.go:356] "Generic (PLEG): container finished" podID="bc39e16b-28fe-413b-bbe0-74418f9a2c0a" containerID="25db5ae48bf1171164d27e10be15c8b3d3f15e91fac2bc84f46bffa06b588622" exitCode=0 Sep 17 15:34:11.671664 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:11.671638 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" event={"ID":"bc39e16b-28fe-413b-bbe0-74418f9a2c0a","Type":"ContainerDied","Data":"25db5ae48bf1171164d27e10be15c8b3d3f15e91fac2bc84f46bffa06b588622"} Sep 17 15:34:11.672050 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:11.671671 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" event={"ID":"bc39e16b-28fe-413b-bbe0-74418f9a2c0a","Type":"ContainerStarted","Data":"ecb7502fd5fe367e99e1eca3450eed57d98a72caa2df1ab453a589dc3c001fb3"} Sep 17 15:34:12.677339 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:12.677260 2581 generic.go:356] "Generic (PLEG): container finished" podID="bc39e16b-28fe-413b-bbe0-74418f9a2c0a" containerID="61c13820130110f4a71f2fa0218be421d8e9fce9d7257d82dbc8f1fac6cddf12" exitCode=0 Sep 17 15:34:12.677668 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:12.677345 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" event={"ID":"bc39e16b-28fe-413b-bbe0-74418f9a2c0a","Type":"ContainerDied","Data":"61c13820130110f4a71f2fa0218be421d8e9fce9d7257d82dbc8f1fac6cddf12"} Sep 17 15:34:13.683636 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:13.683552 2581 generic.go:356] "Generic (PLEG): container finished" podID="bc39e16b-28fe-413b-bbe0-74418f9a2c0a" containerID="6d147dcf86ca68ca3dca3e823f78b6284977e4e22e1c296e85219f562ddd3ac3" exitCode=0 Sep 17 15:34:13.683636 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:13.683604 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" event={"ID":"bc39e16b-28fe-413b-bbe0-74418f9a2c0a","Type":"ContainerDied","Data":"6d147dcf86ca68ca3dca3e823f78b6284977e4e22e1c296e85219f562ddd3ac3"} Sep 17 15:34:14.804634 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:14.804609 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" Sep 17 15:34:14.960606 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:14.960525 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-util\") pod \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\" (UID: \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\") " Sep 17 15:34:14.960606 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:14.960595 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mblr4\" (UniqueName: \"kubernetes.io/projected/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-kube-api-access-mblr4\") pod \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\" (UID: \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\") " Sep 17 15:34:14.960810 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:14.960625 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-bundle\") pod \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\" (UID: \"bc39e16b-28fe-413b-bbe0-74418f9a2c0a\") " Sep 17 15:34:14.961364 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:14.961330 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-bundle" (OuterVolumeSpecName: "bundle") pod "bc39e16b-28fe-413b-bbe0-74418f9a2c0a" (UID: "bc39e16b-28fe-413b-bbe0-74418f9a2c0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:34:14.962633 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:14.962608 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-kube-api-access-mblr4" (OuterVolumeSpecName: "kube-api-access-mblr4") pod "bc39e16b-28fe-413b-bbe0-74418f9a2c0a" (UID: "bc39e16b-28fe-413b-bbe0-74418f9a2c0a"). InnerVolumeSpecName "kube-api-access-mblr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:34:14.966191 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:14.966146 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-util" (OuterVolumeSpecName: "util") pod "bc39e16b-28fe-413b-bbe0-74418f9a2c0a" (UID: "bc39e16b-28fe-413b-bbe0-74418f9a2c0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:34:15.061565 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:15.061537 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mblr4\" (UniqueName: \"kubernetes.io/projected/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-kube-api-access-mblr4\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:34:15.061565 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:15.061560 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-bundle\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:34:15.061565 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:15.061570 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc39e16b-28fe-413b-bbe0-74418f9a2c0a-util\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:34:15.693735 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:15.693704 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" event={"ID":"bc39e16b-28fe-413b-bbe0-74418f9a2c0a","Type":"ContainerDied","Data":"ecb7502fd5fe367e99e1eca3450eed57d98a72caa2df1ab453a589dc3c001fb3"} Sep 17 15:34:15.693869 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:15.693747 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecb7502fd5fe367e99e1eca3450eed57d98a72caa2df1ab453a589dc3c001fb3" Sep 17 15:34:15.693869 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:15.693714 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affppxc" Sep 17 15:34:25.643604 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:25.643575 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:25.783805 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:25.783778 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:26.117176 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:26.117134 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:26.265433 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:26.265409 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:26.728141 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:26.728100 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:27.110823 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.110785 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:27.215003 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.214973 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:27.258364 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.258336 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm"] Sep 17 15:34:27.258623 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.258612 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc39e16b-28fe-413b-bbe0-74418f9a2c0a" containerName="pull" Sep 17 15:34:27.258665 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.258625 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc39e16b-28fe-413b-bbe0-74418f9a2c0a" containerName="pull" Sep 17 15:34:27.258665 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.258634 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc39e16b-28fe-413b-bbe0-74418f9a2c0a" containerName="util" Sep 17 15:34:27.258665 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.258640 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc39e16b-28fe-413b-bbe0-74418f9a2c0a" containerName="util" Sep 17 15:34:27.258665 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.258650 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc39e16b-28fe-413b-bbe0-74418f9a2c0a" containerName="extract" Sep 17 15:34:27.258665 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.258657 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc39e16b-28fe-413b-bbe0-74418f9a2c0a" containerName="extract" Sep 17 15:34:27.258815 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.258706 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc39e16b-28fe-413b-bbe0-74418f9a2c0a" containerName="extract" Sep 17 15:34:27.311541 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.311519 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm"] Sep 17 15:34:27.311662 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.311625 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.315729 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.315708 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 17 15:34:27.315729 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.315711 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 17 15:34:27.316005 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.315940 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 17 15:34:27.316096 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.316026 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 17 15:34:27.316096 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.316034 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-r95mn\"" Sep 17 15:34:27.406301 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.406244 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l"] Sep 17 15:34:27.432955 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.432935 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" Sep 17 15:34:27.435719 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.435689 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 15:34:27.435830 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.435729 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 15:34:27.435893 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.435874 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5lr7h\"" Sep 17 15:34:27.436125 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.436103 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l"] Sep 17 15:34:27.454690 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.454669 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24c16fd2-7c59-480f-a5b1-c52fffe7748d-manifests\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.454796 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.454710 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24c16fd2-7c59-480f-a5b1-c52fffe7748d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.454857 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.454792 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c16fd2-7c59-480f-a5b1-c52fffe7748d-tmp\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.454902 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.454862 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24c16fd2-7c59-480f-a5b1-c52fffe7748d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.454902 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.454894 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl5k4\" (UniqueName: \"kubernetes.io/projected/24c16fd2-7c59-480f-a5b1-c52fffe7748d-kube-api-access-sl5k4\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.460567 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.460539 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:27.555453 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.555426 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l\" (UID: \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" Sep 17 15:34:27.555599 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.555467 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24c16fd2-7c59-480f-a5b1-c52fffe7748d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.555599 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.555487 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrm2f\" (UniqueName: \"kubernetes.io/projected/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-kube-api-access-wrm2f\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l\" (UID: \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" Sep 17 15:34:27.555599 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.555509 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5k4\" (UniqueName: \"kubernetes.io/projected/24c16fd2-7c59-480f-a5b1-c52fffe7748d-kube-api-access-sl5k4\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.555599 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.555542 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24c16fd2-7c59-480f-a5b1-c52fffe7748d-manifests\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.555599 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.555562 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24c16fd2-7c59-480f-a5b1-c52fffe7748d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.555599 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.555586 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l\" (UID: \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" Sep 17 15:34:27.555893 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.555621 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c16fd2-7c59-480f-a5b1-c52fffe7748d-tmp\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.556115 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.556097 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/24c16fd2-7c59-480f-a5b1-c52fffe7748d-manifests\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.557843 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.557823 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c16fd2-7c59-480f-a5b1-c52fffe7748d-tmp\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.557961 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.557904 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24c16fd2-7c59-480f-a5b1-c52fffe7748d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.558067 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.558049 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/24c16fd2-7c59-480f-a5b1-c52fffe7748d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.563790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.563760 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:27.569304 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.569275 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5k4\" (UniqueName: \"kubernetes.io/projected/24c16fd2-7c59-480f-a5b1-c52fffe7748d-kube-api-access-sl5k4\") pod \"opendatahub-operator-controller-manager-5df9996869-cq6xm\" (UID: \"24c16fd2-7c59-480f-a5b1-c52fffe7748d\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.621145 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.621121 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:27.656516 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.656460 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l\" (UID: \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" Sep 17 15:34:27.656616 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.656515 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l\" (UID: \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" Sep 17 15:34:27.656616 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.656557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrm2f\" (UniqueName: \"kubernetes.io/projected/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-kube-api-access-wrm2f\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l\" (UID: \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" Sep 17 15:34:27.656924 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.656889 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l\" (UID: \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" Sep 17 15:34:27.656995 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.656888 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l\" (UID: \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" Sep 17 15:34:27.663951 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.663924 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:27.666044 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.666023 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrm2f\" (UniqueName: \"kubernetes.io/projected/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-kube-api-access-wrm2f\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l\" (UID: \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" Sep 17 15:34:27.743270 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.743244 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" Sep 17 15:34:27.761054 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.761029 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:27.860054 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.860019 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:27.946842 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.946819 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm"] Sep 17 15:34:27.948649 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:34:27.948626 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c16fd2_7c59_480f_a5b1_c52fffe7748d.slice/crio-9f472a6f33992f8f7e22aa6cd634107300abc915c05c1df3bfdc118b34fc3184 WatchSource:0}: Error finding container 9f472a6f33992f8f7e22aa6cd634107300abc915c05c1df3bfdc118b34fc3184: Status 404 returned error can't find the container with id 9f472a6f33992f8f7e22aa6cd634107300abc915c05c1df3bfdc118b34fc3184 Sep 17 15:34:27.984559 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:27.984539 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:28.015565 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:28.015543 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l"] Sep 17 15:34:28.017513 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:34:28.017492 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d9dcd7_33fb_4b46_a3ae_5eb514c38a28.slice/crio-1f3367426079623f8efd2185ee0d41d9ee009446b46ac8630a5ece35947df9ac WatchSource:0}: Error finding container 1f3367426079623f8efd2185ee0d41d9ee009446b46ac8630a5ece35947df9ac: Status 404 returned error can't find the container with id 1f3367426079623f8efd2185ee0d41d9ee009446b46ac8630a5ece35947df9ac Sep 17 15:34:28.749776 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:28.749733 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" event={"ID":"24c16fd2-7c59-480f-a5b1-c52fffe7748d","Type":"ContainerStarted","Data":"9f472a6f33992f8f7e22aa6cd634107300abc915c05c1df3bfdc118b34fc3184"} Sep 17 15:34:28.751651 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:28.751629 2581 generic.go:356] "Generic (PLEG): container finished" podID="13d9dcd7-33fb-4b46-a3ae-5eb514c38a28" containerID="af71907db9cfa1145b73a8f783d029a30baf912e73917f61e8e5c36207ed8fb8" exitCode=0 Sep 17 15:34:28.751769 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:28.751706 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" event={"ID":"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28","Type":"ContainerDied","Data":"af71907db9cfa1145b73a8f783d029a30baf912e73917f61e8e5c36207ed8fb8"} Sep 17 15:34:28.751769 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:28.751742 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" event={"ID":"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28","Type":"ContainerStarted","Data":"1f3367426079623f8efd2185ee0d41d9ee009446b46ac8630a5ece35947df9ac"} Sep 17 15:34:29.758461 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:29.758387 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" event={"ID":"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28","Type":"ContainerStarted","Data":"4c6251677026cf3bdc4665426fe0037f787afde4c5b78904933bd8621353154a"} Sep 17 15:34:31.769240 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:31.769196 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" event={"ID":"24c16fd2-7c59-480f-a5b1-c52fffe7748d","Type":"ContainerStarted","Data":"7085a30ec94ab4958573bafcdfaa66de6f22ef06e47a5c9da004da932b33a10f"} Sep 17 15:34:31.771046 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:31.771021 2581 generic.go:356] "Generic (PLEG): container finished" podID="13d9dcd7-33fb-4b46-a3ae-5eb514c38a28" containerID="4c6251677026cf3bdc4665426fe0037f787afde4c5b78904933bd8621353154a" exitCode=0 Sep 17 15:34:31.771157 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:31.771074 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" event={"ID":"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28","Type":"ContainerDied","Data":"4c6251677026cf3bdc4665426fe0037f787afde4c5b78904933bd8621353154a"} Sep 17 15:34:32.776798 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:32.776766 2581 generic.go:356] "Generic (PLEG): container finished" podID="24c16fd2-7c59-480f-a5b1-c52fffe7748d" containerID="7085a30ec94ab4958573bafcdfaa66de6f22ef06e47a5c9da004da932b33a10f" exitCode=0 Sep 17 15:34:32.777215 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:32.776854 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" event={"ID":"24c16fd2-7c59-480f-a5b1-c52fffe7748d","Type":"ContainerDied","Data":"7085a30ec94ab4958573bafcdfaa66de6f22ef06e47a5c9da004da932b33a10f"} Sep 17 15:34:32.779249 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:32.779225 2581 generic.go:356] "Generic (PLEG): container finished" podID="13d9dcd7-33fb-4b46-a3ae-5eb514c38a28" containerID="3e9f578959a9327d1508b4c4a7a26d877028eb944513e7c1ff88969f51e16712" exitCode=0 Sep 17 15:34:32.779364 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:32.779267 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" event={"ID":"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28","Type":"ContainerDied","Data":"3e9f578959a9327d1508b4c4a7a26d877028eb944513e7c1ff88969f51e16712"} Sep 17 15:34:33.784824 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:33.784790 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" event={"ID":"24c16fd2-7c59-480f-a5b1-c52fffe7748d","Type":"ContainerStarted","Data":"224d3a706b1bbef41395a483afdab75a79d76f7abc5c73b4b050ed4ecf238901"} Sep 17 15:34:33.809620 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:33.809555 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" podStartSLOduration=2.166564754 podStartE2EDuration="6.809535115s" totalImagesPullingTime="3.563264341s" totalInitContainerRuntime="1.07970602s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:34:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:34:33.806770409 +0000 UTC m=+260.803007157" watchObservedRunningTime="2026-09-17 15:34:33.809535115 +0000 UTC m=+260.805771864" Sep 17 15:34:33.902301 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:33.902275 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" Sep 17 15:34:34.002768 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.002738 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrm2f\" (UniqueName: \"kubernetes.io/projected/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-kube-api-access-wrm2f\") pod \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\" (UID: \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\") " Sep 17 15:34:34.002768 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.002772 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-bundle\") pod \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\" (UID: \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\") " Sep 17 15:34:34.002988 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.002820 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-util\") pod \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\" (UID: \"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28\") " Sep 17 15:34:34.003516 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.003490 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-bundle" (OuterVolumeSpecName: "bundle") pod "13d9dcd7-33fb-4b46-a3ae-5eb514c38a28" (UID: "13d9dcd7-33fb-4b46-a3ae-5eb514c38a28"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:34:34.004790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.004759 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-kube-api-access-wrm2f" (OuterVolumeSpecName: "kube-api-access-wrm2f") pod "13d9dcd7-33fb-4b46-a3ae-5eb514c38a28" (UID: "13d9dcd7-33fb-4b46-a3ae-5eb514c38a28"). InnerVolumeSpecName "kube-api-access-wrm2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:34:34.008242 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.008214 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-util" (OuterVolumeSpecName: "util") pod "13d9dcd7-33fb-4b46-a3ae-5eb514c38a28" (UID: "13d9dcd7-33fb-4b46-a3ae-5eb514c38a28"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:34:34.103838 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.103786 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-util\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:34:34.103838 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.103804 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrm2f\" (UniqueName: \"kubernetes.io/projected/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-kube-api-access-wrm2f\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:34:34.103838 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.103814 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/13d9dcd7-33fb-4b46-a3ae-5eb514c38a28-bundle\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:34:34.641156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.641102 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:34.791485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.791455 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" Sep 17 15:34:34.791485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.791477 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0jhl7l" event={"ID":"13d9dcd7-33fb-4b46-a3ae-5eb514c38a28","Type":"ContainerDied","Data":"1f3367426079623f8efd2185ee0d41d9ee009446b46ac8630a5ece35947df9ac"} Sep 17 15:34:34.791932 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.791545 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f3367426079623f8efd2185ee0d41d9ee009446b46ac8630a5ece35947df9ac" Sep 17 15:34:34.791932 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.791866 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:34.813861 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:34.813841 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:39.178735 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:39.178663 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:39.278793 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:39.278756 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:45.798926 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:45.798881 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-cq6xm" Sep 17 15:34:47.549014 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:47.548985 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:47.673057 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:47.673022 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:48.004253 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.004181 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j"] Sep 17 15:34:48.004483 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.004470 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d9dcd7-33fb-4b46-a3ae-5eb514c38a28" containerName="pull" Sep 17 15:34:48.004526 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.004485 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d9dcd7-33fb-4b46-a3ae-5eb514c38a28" containerName="pull" Sep 17 15:34:48.004526 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.004506 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d9dcd7-33fb-4b46-a3ae-5eb514c38a28" containerName="extract" Sep 17 15:34:48.004526 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.004512 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d9dcd7-33fb-4b46-a3ae-5eb514c38a28" containerName="extract" Sep 17 15:34:48.004526 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.004522 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d9dcd7-33fb-4b46-a3ae-5eb514c38a28" containerName="util" Sep 17 15:34:48.004643 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.004528 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d9dcd7-33fb-4b46-a3ae-5eb514c38a28" containerName="util" Sep 17 15:34:48.004643 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.004574 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d9dcd7-33fb-4b46-a3ae-5eb514c38a28" containerName="extract" Sep 17 15:34:48.011294 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.011272 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" Sep 17 15:34:48.014249 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.014224 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 17 15:34:48.014423 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.014409 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 15:34:48.015091 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.015072 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 15:34:48.015156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.015116 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6l7xx\"" Sep 17 15:34:48.018226 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.018211 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 17 15:34:48.023127 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.023105 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:48.036512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.036486 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j"] Sep 17 15:34:48.107243 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.107213 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70d3c6a-33cc-465c-996f-96e2608fee43-tls-certs\") pod \"kube-auth-proxy-5dc8ddcd98-zgj6j\" (UID: \"c70d3c6a-33cc-465c-996f-96e2608fee43\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" Sep 17 15:34:48.107357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.107256 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgxmc\" (UniqueName: \"kubernetes.io/projected/c70d3c6a-33cc-465c-996f-96e2608fee43-kube-api-access-jgxmc\") pod \"kube-auth-proxy-5dc8ddcd98-zgj6j\" (UID: \"c70d3c6a-33cc-465c-996f-96e2608fee43\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" Sep 17 15:34:48.107357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.107318 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c70d3c6a-33cc-465c-996f-96e2608fee43-tmp\") pod \"kube-auth-proxy-5dc8ddcd98-zgj6j\" (UID: \"c70d3c6a-33cc-465c-996f-96e2608fee43\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" Sep 17 15:34:48.152442 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.152402 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:48.208601 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.208572 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70d3c6a-33cc-465c-996f-96e2608fee43-tls-certs\") pod \"kube-auth-proxy-5dc8ddcd98-zgj6j\" (UID: \"c70d3c6a-33cc-465c-996f-96e2608fee43\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" Sep 17 15:34:48.208699 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.208610 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgxmc\" (UniqueName: \"kubernetes.io/projected/c70d3c6a-33cc-465c-996f-96e2608fee43-kube-api-access-jgxmc\") pod \"kube-auth-proxy-5dc8ddcd98-zgj6j\" (UID: \"c70d3c6a-33cc-465c-996f-96e2608fee43\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" Sep 17 15:34:48.208768 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.208730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c70d3c6a-33cc-465c-996f-96e2608fee43-tmp\") pod \"kube-auth-proxy-5dc8ddcd98-zgj6j\" (UID: \"c70d3c6a-33cc-465c-996f-96e2608fee43\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" Sep 17 15:34:48.210814 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.210789 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c70d3c6a-33cc-465c-996f-96e2608fee43-tmp\") pod \"kube-auth-proxy-5dc8ddcd98-zgj6j\" (UID: \"c70d3c6a-33cc-465c-996f-96e2608fee43\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" Sep 17 15:34:48.211030 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.211014 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70d3c6a-33cc-465c-996f-96e2608fee43-tls-certs\") pod \"kube-auth-proxy-5dc8ddcd98-zgj6j\" (UID: \"c70d3c6a-33cc-465c-996f-96e2608fee43\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" Sep 17 15:34:48.224340 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.224312 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgxmc\" (UniqueName: \"kubernetes.io/projected/c70d3c6a-33cc-465c-996f-96e2608fee43-kube-api-access-jgxmc\") pod \"kube-auth-proxy-5dc8ddcd98-zgj6j\" (UID: \"c70d3c6a-33cc-465c-996f-96e2608fee43\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" Sep 17 15:34:48.320842 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.320818 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" Sep 17 15:34:48.597022 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.597001 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j"] Sep 17 15:34:48.599008 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:34:48.598977 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70d3c6a_33cc_465c_996f_96e2608fee43.slice/crio-22bd143f6051c596094b0f8603bab893ca7b4aa5b84d517134825ea398d3af7e WatchSource:0}: Error finding container 22bd143f6051c596094b0f8603bab893ca7b4aa5b84d517134825ea398d3af7e: Status 404 returned error can't find the container with id 22bd143f6051c596094b0f8603bab893ca7b4aa5b84d517134825ea398d3af7e Sep 17 15:34:48.854484 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:48.854400 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" event={"ID":"c70d3c6a-33cc-465c-996f-96e2608fee43","Type":"ContainerStarted","Data":"22bd143f6051c596094b0f8603bab893ca7b4aa5b84d517134825ea398d3af7e"} Sep 17 15:34:49.400269 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.400232 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:49.412546 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.412517 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb"] Sep 17 15:34:49.416192 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.416171 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.418525 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.418499 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 17 15:34:49.418643 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.418601 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 17 15:34:49.419010 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.418721 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 17 15:34:49.419010 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.418824 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2964l\"" Sep 17 15:34:49.419010 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.418901 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 15:34:49.434323 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.434292 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb"] Sep 17 15:34:49.518928 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.518877 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7xx\" (UniqueName: \"kubernetes.io/projected/fe177fc4-e121-404c-aa44-15d095d3890d-kube-api-access-5p7xx\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.519114 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.518947 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.519114 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.518984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.519114 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.519085 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe177fc4-e121-404c-aa44-15d095d3890d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.519275 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.519142 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe177fc4-e121-404c-aa44-15d095d3890d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.519275 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.519181 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.519275 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.519229 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe177fc4-e121-404c-aa44-15d095d3890d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.526244 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.526215 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:49.619944 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.619899 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe177fc4-e121-404c-aa44-15d095d3890d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.620492 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.619969 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7xx\" (UniqueName: \"kubernetes.io/projected/fe177fc4-e121-404c-aa44-15d095d3890d-kube-api-access-5p7xx\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.620492 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.620003 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.620492 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.620045 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.620492 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.620093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe177fc4-e121-404c-aa44-15d095d3890d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.620492 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.620125 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe177fc4-e121-404c-aa44-15d095d3890d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.620492 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.620160 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.620871 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.620840 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe177fc4-e121-404c-aa44-15d095d3890d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.623131 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.623093 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.623244 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.623132 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.623432 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.623398 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.623858 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.623837 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe177fc4-e121-404c-aa44-15d095d3890d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.629958 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.629900 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe177fc4-e121-404c-aa44-15d095d3890d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.633651 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.633627 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7xx\" (UniqueName: \"kubernetes.io/projected/fe177fc4-e121-404c-aa44-15d095d3890d-kube-api-access-5p7xx\") pod \"istiod-openshift-gateway-7b7995d5cd-wwmzb\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:49.729739 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:49.729651 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:50.013201 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:50.013148 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb"] Sep 17 15:34:50.015937 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:34:50.015886 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe177fc4_e121_404c_aa44_15d095d3890d.slice/crio-dde26c73bb77b20e94744c031eab987032c7fa3fb785578bb1395026571728fd WatchSource:0}: Error finding container dde26c73bb77b20e94744c031eab987032c7fa3fb785578bb1395026571728fd: Status 404 returned error can't find the container with id dde26c73bb77b20e94744c031eab987032c7fa3fb785578bb1395026571728fd Sep 17 15:34:50.533791 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:50.533758 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:50.865829 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:50.865747 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" event={"ID":"fe177fc4-e121-404c-aa44-15d095d3890d","Type":"ContainerStarted","Data":"dde26c73bb77b20e94744c031eab987032c7fa3fb785578bb1395026571728fd"} Sep 17 15:34:51.873522 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:51.873488 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" event={"ID":"c70d3c6a-33cc-465c-996f-96e2608fee43","Type":"ContainerStarted","Data":"9c13971e742f6a841afa2608c9c03de16c90247a93bf5659fc2859044a47f86b"} Sep 17 15:34:51.896464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:51.893147 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-zgj6j" podStartSLOduration=2.036330691 podStartE2EDuration="4.893125455s" totalImagesPullingTime="2.856794764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:34:51.891330663 +0000 UTC m=+278.887567411" watchObservedRunningTime="2026-09-17 15:34:51.893125455 +0000 UTC m=+278.889362201" Sep 17 15:34:53.184711 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:53.184664 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:34:53.184962 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:53.184753 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:34:53.883970 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:53.883934 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" event={"ID":"fe177fc4-e121-404c-aa44-15d095d3890d","Type":"ContainerStarted","Data":"31ade322094afb94fb4486e6fe22e18d350319977952d7316ce113f94db74565"} Sep 17 15:34:53.884162 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:53.884065 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:53.904826 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:53.904757 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" podStartSLOduration=1.738935901 podStartE2EDuration="4.904738666s" totalImagesPullingTime="3.165802765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:34:53.904163069 +0000 UTC m=+280.900399817" watchObservedRunningTime="2026-09-17 15:34:53.904738666 +0000 UTC m=+280.900975413" Sep 17 15:34:53.994717 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:53.994681 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:54.889859 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:54.889828 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:34:55.405114 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:55.405059 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:58.809966 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:58.809930 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:58.963342 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:58.963316 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:59.067017 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:59.066940 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:59.166383 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:59.166353 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:59.382597 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:59.382523 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:34:59.595664 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:34:59.595630 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:35:06.112300 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.112259 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84"] Sep 17 15:35:06.115241 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.115224 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.118448 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.118344 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-75rsx\"" Sep 17 15:35:06.118448 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.118365 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 17 15:35:06.118448 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.118344 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 17 15:35:06.118448 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.118423 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 17 15:35:06.118448 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.118425 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 17 15:35:06.118767 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.118373 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 17 15:35:06.128610 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.128585 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84"] Sep 17 15:35:06.260173 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.260142 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ac1f6b4-4924-4b0f-9069-1bbabcfda87c-metrics-cert\") pod \"lws-controller-manager-6c7d57fb89-wfn84\" (UID: \"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.260338 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.260192 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnk68\" (UniqueName: \"kubernetes.io/projected/3ac1f6b4-4924-4b0f-9069-1bbabcfda87c-kube-api-access-wnk68\") pod \"lws-controller-manager-6c7d57fb89-wfn84\" (UID: \"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.260338 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.260218 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ac1f6b4-4924-4b0f-9069-1bbabcfda87c-cert\") pod \"lws-controller-manager-6c7d57fb89-wfn84\" (UID: \"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.260338 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.260285 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3ac1f6b4-4924-4b0f-9069-1bbabcfda87c-manager-config\") pod \"lws-controller-manager-6c7d57fb89-wfn84\" (UID: \"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.361602 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.361568 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3ac1f6b4-4924-4b0f-9069-1bbabcfda87c-manager-config\") pod \"lws-controller-manager-6c7d57fb89-wfn84\" (UID: \"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.361753 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.361614 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ac1f6b4-4924-4b0f-9069-1bbabcfda87c-metrics-cert\") pod \"lws-controller-manager-6c7d57fb89-wfn84\" (UID: \"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.361753 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.361659 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnk68\" (UniqueName: \"kubernetes.io/projected/3ac1f6b4-4924-4b0f-9069-1bbabcfda87c-kube-api-access-wnk68\") pod \"lws-controller-manager-6c7d57fb89-wfn84\" (UID: \"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.361753 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.361699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ac1f6b4-4924-4b0f-9069-1bbabcfda87c-cert\") pod \"lws-controller-manager-6c7d57fb89-wfn84\" (UID: \"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.362242 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.362221 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3ac1f6b4-4924-4b0f-9069-1bbabcfda87c-manager-config\") pod \"lws-controller-manager-6c7d57fb89-wfn84\" (UID: \"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.364199 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.364149 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ac1f6b4-4924-4b0f-9069-1bbabcfda87c-metrics-cert\") pod \"lws-controller-manager-6c7d57fb89-wfn84\" (UID: \"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.364199 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.364154 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ac1f6b4-4924-4b0f-9069-1bbabcfda87c-cert\") pod \"lws-controller-manager-6c7d57fb89-wfn84\" (UID: \"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.371023 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.370996 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnk68\" (UniqueName: \"kubernetes.io/projected/3ac1f6b4-4924-4b0f-9069-1bbabcfda87c-kube-api-access-wnk68\") pod \"lws-controller-manager-6c7d57fb89-wfn84\" (UID: \"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c\") " pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.424894 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.424871 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:06.703726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.703649 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84"] Sep 17 15:35:06.707367 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:35:06.707342 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac1f6b4_4924_4b0f_9069_1bbabcfda87c.slice/crio-6009e2aa983ea592671d401bbae50e05f73f58064fd1861b52d65e5e60750e24 WatchSource:0}: Error finding container 6009e2aa983ea592671d401bbae50e05f73f58064fd1861b52d65e5e60750e24: Status 404 returned error can't find the container with id 6009e2aa983ea592671d401bbae50e05f73f58064fd1861b52d65e5e60750e24 Sep 17 15:35:06.938757 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:06.938724 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" event={"ID":"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c","Type":"ContainerStarted","Data":"6009e2aa983ea592671d401bbae50e05f73f58064fd1861b52d65e5e60750e24"} Sep 17 15:35:08.590096 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:08.590067 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:35:08.727879 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:08.727810 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:35:08.949826 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:08.949790 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" event={"ID":"3ac1f6b4-4924-4b0f-9069-1bbabcfda87c","Type":"ContainerStarted","Data":"66c7a2912fcab7c0c141b77806695549a595156403e7318db9e503d4263828c2"} Sep 17 15:35:08.950026 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:08.949902 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:08.993953 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:08.993864 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:35:09.140424 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:09.140397 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:35:10.090487 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:10.090454 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:35:10.235286 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:10.235250 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:35:10.911083 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:10.911035 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" podStartSLOduration=3.225472213 podStartE2EDuration="4.911018384s" totalImagesPullingTime="1.685546171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:35:08.973822694 +0000 UTC m=+295.970059439" watchObservedRunningTime="2026-09-17 15:35:10.911018384 +0000 UTC m=+297.907255131" Sep 17 15:35:10.912565 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:10.912545 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb"] Sep 17 15:35:10.912768 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:10.912749 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" podUID="fe177fc4-e121-404c-aa44-15d095d3890d" containerName="discovery" containerID="cri-o://31ade322094afb94fb4486e6fe22e18d350319977952d7316ce113f94db74565" gracePeriod=30 Sep 17 15:35:11.155622 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.155600 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:35:11.303649 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.303618 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe177fc4-e121-404c-aa44-15d095d3890d-istio-token\") pod \"fe177fc4-e121-404c-aa44-15d095d3890d\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " Sep 17 15:35:11.303805 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.303658 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p7xx\" (UniqueName: \"kubernetes.io/projected/fe177fc4-e121-404c-aa44-15d095d3890d-kube-api-access-5p7xx\") pod \"fe177fc4-e121-404c-aa44-15d095d3890d\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " Sep 17 15:35:11.303805 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.303687 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-cacerts\") pod \"fe177fc4-e121-404c-aa44-15d095d3890d\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " Sep 17 15:35:11.303805 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.303711 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-istio-kubeconfig\") pod \"fe177fc4-e121-404c-aa44-15d095d3890d\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " Sep 17 15:35:11.303805 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.303733 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-istio-csr-dns-cert\") pod \"fe177fc4-e121-404c-aa44-15d095d3890d\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " Sep 17 15:35:11.303805 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.303758 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe177fc4-e121-404c-aa44-15d095d3890d-istio-csr-ca-configmap\") pod \"fe177fc4-e121-404c-aa44-15d095d3890d\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " Sep 17 15:35:11.303805 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.303794 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe177fc4-e121-404c-aa44-15d095d3890d-local-certs\") pod \"fe177fc4-e121-404c-aa44-15d095d3890d\" (UID: \"fe177fc4-e121-404c-aa44-15d095d3890d\") " Sep 17 15:35:11.304294 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.304227 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe177fc4-e121-404c-aa44-15d095d3890d-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "fe177fc4-e121-404c-aa44-15d095d3890d" (UID: "fe177fc4-e121-404c-aa44-15d095d3890d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:35:11.306149 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.306120 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-cacerts" (OuterVolumeSpecName: "cacerts") pod "fe177fc4-e121-404c-aa44-15d095d3890d" (UID: "fe177fc4-e121-404c-aa44-15d095d3890d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:35:11.306272 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.306247 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "fe177fc4-e121-404c-aa44-15d095d3890d" (UID: "fe177fc4-e121-404c-aa44-15d095d3890d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:35:11.306391 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.306352 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe177fc4-e121-404c-aa44-15d095d3890d-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe177fc4-e121-404c-aa44-15d095d3890d" (UID: "fe177fc4-e121-404c-aa44-15d095d3890d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:35:11.306460 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.306429 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "fe177fc4-e121-404c-aa44-15d095d3890d" (UID: "fe177fc4-e121-404c-aa44-15d095d3890d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:35:11.306460 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.306438 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe177fc4-e121-404c-aa44-15d095d3890d-local-certs" (OuterVolumeSpecName: "local-certs") pod "fe177fc4-e121-404c-aa44-15d095d3890d" (UID: "fe177fc4-e121-404c-aa44-15d095d3890d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:35:11.306561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.306530 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe177fc4-e121-404c-aa44-15d095d3890d-kube-api-access-5p7xx" (OuterVolumeSpecName: "kube-api-access-5p7xx") pod "fe177fc4-e121-404c-aa44-15d095d3890d" (UID: "fe177fc4-e121-404c-aa44-15d095d3890d"). InnerVolumeSpecName "kube-api-access-5p7xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:35:11.405142 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.405114 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe177fc4-e121-404c-aa44-15d095d3890d-istio-token\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:35:11.405142 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.405134 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p7xx\" (UniqueName: \"kubernetes.io/projected/fe177fc4-e121-404c-aa44-15d095d3890d-kube-api-access-5p7xx\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:35:11.405142 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.405142 2581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-cacerts\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:35:11.405305 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.405152 2581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-istio-kubeconfig\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:35:11.405305 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.405161 2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe177fc4-e121-404c-aa44-15d095d3890d-istio-csr-dns-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:35:11.405305 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.405169 2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe177fc4-e121-404c-aa44-15d095d3890d-istio-csr-ca-configmap\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:35:11.405305 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.405178 2581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe177fc4-e121-404c-aa44-15d095d3890d-local-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:35:11.967792 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.967752 2581 generic.go:356] "Generic (PLEG): container finished" podID="fe177fc4-e121-404c-aa44-15d095d3890d" containerID="31ade322094afb94fb4486e6fe22e18d350319977952d7316ce113f94db74565" exitCode=0 Sep 17 15:35:11.967975 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.967802 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" Sep 17 15:35:11.967975 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.967832 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" event={"ID":"fe177fc4-e121-404c-aa44-15d095d3890d","Type":"ContainerDied","Data":"31ade322094afb94fb4486e6fe22e18d350319977952d7316ce113f94db74565"} Sep 17 15:35:11.967975 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.967885 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb" event={"ID":"fe177fc4-e121-404c-aa44-15d095d3890d","Type":"ContainerDied","Data":"dde26c73bb77b20e94744c031eab987032c7fa3fb785578bb1395026571728fd"} Sep 17 15:35:11.967975 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.967905 2581 scope.go:117] "RemoveContainer" containerID="31ade322094afb94fb4486e6fe22e18d350319977952d7316ce113f94db74565" Sep 17 15:35:11.976455 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.976438 2581 scope.go:117] "RemoveContainer" containerID="31ade322094afb94fb4486e6fe22e18d350319977952d7316ce113f94db74565" Sep 17 15:35:11.976728 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:35:11.976711 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ade322094afb94fb4486e6fe22e18d350319977952d7316ce113f94db74565\": container with ID starting with 31ade322094afb94fb4486e6fe22e18d350319977952d7316ce113f94db74565 not found: ID does not exist" containerID="31ade322094afb94fb4486e6fe22e18d350319977952d7316ce113f94db74565" Sep 17 15:35:11.976791 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.976739 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ade322094afb94fb4486e6fe22e18d350319977952d7316ce113f94db74565"} err="failed to get container status \"31ade322094afb94fb4486e6fe22e18d350319977952d7316ce113f94db74565\": rpc error: code = NotFound desc = could not find container \"31ade322094afb94fb4486e6fe22e18d350319977952d7316ce113f94db74565\": container with ID starting with 31ade322094afb94fb4486e6fe22e18d350319977952d7316ce113f94db74565 not found: ID does not exist" Sep 17 15:35:11.985923 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.985894 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb"] Sep 17 15:35:11.992799 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:11.992779 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wwmzb"] Sep 17 15:35:13.363506 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:13.363482 2581 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 15:35:13.462189 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:13.462042 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe177fc4-e121-404c-aa44-15d095d3890d" path="/var/lib/kubelet/pods/fe177fc4-e121-404c-aa44-15d095d3890d/volumes" Sep 17 15:35:19.959703 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:19.959669 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c7d57fb89-wfn84" Sep 17 15:35:59.856412 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:59.856376 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:35:59.966052 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:35:59.966002 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:04.375075 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.375043 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b"] Sep 17 15:36:04.375444 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.375333 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe177fc4-e121-404c-aa44-15d095d3890d" containerName="discovery" Sep 17 15:36:04.375444 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.375344 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe177fc4-e121-404c-aa44-15d095d3890d" containerName="discovery" Sep 17 15:36:04.375444 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.375410 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe177fc4-e121-404c-aa44-15d095d3890d" containerName="discovery" Sep 17 15:36:04.379152 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.379135 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" Sep 17 15:36:04.381743 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.381719 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 15:36:04.381985 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.381972 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5lr7h\"" Sep 17 15:36:04.382496 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.382482 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 15:36:04.395610 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.395588 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b"] Sep 17 15:36:04.468258 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.468231 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph"] Sep 17 15:36:04.471876 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.471862 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" Sep 17 15:36:04.491110 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.491083 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph"] Sep 17 15:36:04.515455 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.515425 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph\" (UID: \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" Sep 17 15:36:04.515558 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.515487 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e09031a0-e098-485a-ac1e-99a82fffc196-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b\" (UID: \"e09031a0-e098-485a-ac1e-99a82fffc196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" Sep 17 15:36:04.515558 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.515506 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph\" (UID: \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" Sep 17 15:36:04.515558 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.515532 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e09031a0-e098-485a-ac1e-99a82fffc196-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b\" (UID: \"e09031a0-e098-485a-ac1e-99a82fffc196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" Sep 17 15:36:04.515658 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.515575 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8754g\" (UniqueName: \"kubernetes.io/projected/e09031a0-e098-485a-ac1e-99a82fffc196-kube-api-access-8754g\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b\" (UID: \"e09031a0-e098-485a-ac1e-99a82fffc196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" Sep 17 15:36:04.515658 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.515600 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbw7m\" (UniqueName: \"kubernetes.io/projected/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-kube-api-access-sbw7m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph\" (UID: \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" Sep 17 15:36:04.549851 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.549827 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs"] Sep 17 15:36:04.553640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.553625 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" Sep 17 15:36:04.563795 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.563774 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs"] Sep 17 15:36:04.616421 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.616396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e09031a0-e098-485a-ac1e-99a82fffc196-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b\" (UID: \"e09031a0-e098-485a-ac1e-99a82fffc196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" Sep 17 15:36:04.616568 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.616432 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8754g\" (UniqueName: \"kubernetes.io/projected/e09031a0-e098-485a-ac1e-99a82fffc196-kube-api-access-8754g\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b\" (UID: \"e09031a0-e098-485a-ac1e-99a82fffc196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" Sep 17 15:36:04.616568 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.616453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbw7m\" (UniqueName: \"kubernetes.io/projected/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-kube-api-access-sbw7m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph\" (UID: \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" Sep 17 15:36:04.616568 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.616483 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz4bn\" (UniqueName: \"kubernetes.io/projected/f4efa66a-d1af-479f-9972-e513d718ecbe-kube-api-access-rz4bn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs\" (UID: \"f4efa66a-d1af-479f-9972-e513d718ecbe\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" Sep 17 15:36:04.616568 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.616503 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph\" (UID: \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" Sep 17 15:36:04.616568 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.616533 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4efa66a-d1af-479f-9972-e513d718ecbe-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs\" (UID: \"f4efa66a-d1af-479f-9972-e513d718ecbe\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" Sep 17 15:36:04.616870 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.616599 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4efa66a-d1af-479f-9972-e513d718ecbe-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs\" (UID: \"f4efa66a-d1af-479f-9972-e513d718ecbe\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" Sep 17 15:36:04.616870 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.616639 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e09031a0-e098-485a-ac1e-99a82fffc196-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b\" (UID: \"e09031a0-e098-485a-ac1e-99a82fffc196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" Sep 17 15:36:04.616870 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.616660 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph\" (UID: \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" Sep 17 15:36:04.616870 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.616839 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e09031a0-e098-485a-ac1e-99a82fffc196-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b\" (UID: \"e09031a0-e098-485a-ac1e-99a82fffc196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" Sep 17 15:36:04.617077 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.616891 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph\" (UID: \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" Sep 17 15:36:04.617077 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.616902 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e09031a0-e098-485a-ac1e-99a82fffc196-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b\" (UID: \"e09031a0-e098-485a-ac1e-99a82fffc196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" Sep 17 15:36:04.617077 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.616968 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph\" (UID: \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" Sep 17 15:36:04.631134 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.631072 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8754g\" (UniqueName: \"kubernetes.io/projected/e09031a0-e098-485a-ac1e-99a82fffc196-kube-api-access-8754g\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b\" (UID: \"e09031a0-e098-485a-ac1e-99a82fffc196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" Sep 17 15:36:04.631305 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.631284 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbw7m\" (UniqueName: \"kubernetes.io/projected/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-kube-api-access-sbw7m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph\" (UID: \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" Sep 17 15:36:04.656157 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.656131 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks"] Sep 17 15:36:04.667813 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.667791 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" Sep 17 15:36:04.678655 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.678633 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks"] Sep 17 15:36:04.688180 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.688161 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" Sep 17 15:36:04.717227 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.717201 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4efa66a-d1af-479f-9972-e513d718ecbe-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs\" (UID: \"f4efa66a-d1af-479f-9972-e513d718ecbe\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" Sep 17 15:36:04.717357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.717238 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/881dc300-89e8-4a8f-a39a-2402449dd192-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks\" (UID: \"881dc300-89e8-4a8f-a39a-2402449dd192\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" Sep 17 15:36:04.717357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.717262 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4efa66a-d1af-479f-9972-e513d718ecbe-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs\" (UID: \"f4efa66a-d1af-479f-9972-e513d718ecbe\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" Sep 17 15:36:04.717357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.717336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzqbl\" (UniqueName: \"kubernetes.io/projected/881dc300-89e8-4a8f-a39a-2402449dd192-kube-api-access-gzqbl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks\" (UID: \"881dc300-89e8-4a8f-a39a-2402449dd192\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" Sep 17 15:36:04.717357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.717355 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz4bn\" (UniqueName: \"kubernetes.io/projected/f4efa66a-d1af-479f-9972-e513d718ecbe-kube-api-access-rz4bn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs\" (UID: \"f4efa66a-d1af-479f-9972-e513d718ecbe\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" Sep 17 15:36:04.717519 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.717391 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/881dc300-89e8-4a8f-a39a-2402449dd192-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks\" (UID: \"881dc300-89e8-4a8f-a39a-2402449dd192\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" Sep 17 15:36:04.717640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.717617 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4efa66a-d1af-479f-9972-e513d718ecbe-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs\" (UID: \"f4efa66a-d1af-479f-9972-e513d718ecbe\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" Sep 17 15:36:04.717683 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.717642 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4efa66a-d1af-479f-9972-e513d718ecbe-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs\" (UID: \"f4efa66a-d1af-479f-9972-e513d718ecbe\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" Sep 17 15:36:04.736840 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.735728 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz4bn\" (UniqueName: \"kubernetes.io/projected/f4efa66a-d1af-479f-9972-e513d718ecbe-kube-api-access-rz4bn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs\" (UID: \"f4efa66a-d1af-479f-9972-e513d718ecbe\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" Sep 17 15:36:04.784344 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.784312 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" Sep 17 15:36:04.818458 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.818426 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzqbl\" (UniqueName: \"kubernetes.io/projected/881dc300-89e8-4a8f-a39a-2402449dd192-kube-api-access-gzqbl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks\" (UID: \"881dc300-89e8-4a8f-a39a-2402449dd192\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" Sep 17 15:36:04.818615 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.818466 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/881dc300-89e8-4a8f-a39a-2402449dd192-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks\" (UID: \"881dc300-89e8-4a8f-a39a-2402449dd192\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" Sep 17 15:36:04.818615 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.818492 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/881dc300-89e8-4a8f-a39a-2402449dd192-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks\" (UID: \"881dc300-89e8-4a8f-a39a-2402449dd192\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" Sep 17 15:36:04.818769 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.818747 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/881dc300-89e8-4a8f-a39a-2402449dd192-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks\" (UID: \"881dc300-89e8-4a8f-a39a-2402449dd192\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" Sep 17 15:36:04.818828 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.818772 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/881dc300-89e8-4a8f-a39a-2402449dd192-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks\" (UID: \"881dc300-89e8-4a8f-a39a-2402449dd192\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" Sep 17 15:36:04.833318 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.833286 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzqbl\" (UniqueName: \"kubernetes.io/projected/881dc300-89e8-4a8f-a39a-2402449dd192-kube-api-access-gzqbl\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks\" (UID: \"881dc300-89e8-4a8f-a39a-2402449dd192\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" Sep 17 15:36:04.862060 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.862036 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" Sep 17 15:36:04.973999 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.973974 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b"] Sep 17 15:36:04.975992 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:36:04.975964 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09031a0_e098_485a_ac1e_99a82fffc196.slice/crio-60a18471449f61fad8babc0451b924e95d7354207495c6b7c7206090de8641a7 WatchSource:0}: Error finding container 60a18471449f61fad8babc0451b924e95d7354207495c6b7c7206090de8641a7: Status 404 returned error can't find the container with id 60a18471449f61fad8babc0451b924e95d7354207495c6b7c7206090de8641a7 Sep 17 15:36:04.976746 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:04.976732 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" Sep 17 15:36:05.066629 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:05.066603 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph"] Sep 17 15:36:05.068886 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:36:05.068858 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9a76a5_c8aa_45d1_a559_f82a39a2f305.slice/crio-8f8e230f5d314c4222cec460e6dd9a005985bea4bb9dc9e379d29e19819c8b7a WatchSource:0}: Error finding container 8f8e230f5d314c4222cec460e6dd9a005985bea4bb9dc9e379d29e19819c8b7a: Status 404 returned error can't find the container with id 8f8e230f5d314c4222cec460e6dd9a005985bea4bb9dc9e379d29e19819c8b7a Sep 17 15:36:05.141786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:05.141760 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs"] Sep 17 15:36:05.160284 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:36:05.160261 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4efa66a_d1af_479f_9972_e513d718ecbe.slice/crio-cbc00837ecd4f45905d8d6b06bc50ea9d58636b406bc94c1fa686b6c1dce6211 WatchSource:0}: Error finding container cbc00837ecd4f45905d8d6b06bc50ea9d58636b406bc94c1fa686b6c1dce6211: Status 404 returned error can't find the container with id cbc00837ecd4f45905d8d6b06bc50ea9d58636b406bc94c1fa686b6c1dce6211 Sep 17 15:36:05.204008 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:05.203972 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" event={"ID":"0a9a76a5-c8aa-45d1-a559-f82a39a2f305","Type":"ContainerStarted","Data":"752fc570271eb09adef63dd8dfe40e4c634be5977f21beaa4b0b21de515dc374"} Sep 17 15:36:05.204126 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:05.204030 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" event={"ID":"0a9a76a5-c8aa-45d1-a559-f82a39a2f305","Type":"ContainerStarted","Data":"8f8e230f5d314c4222cec460e6dd9a005985bea4bb9dc9e379d29e19819c8b7a"} Sep 17 15:36:05.205650 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:05.205619 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" event={"ID":"f4efa66a-d1af-479f-9972-e513d718ecbe","Type":"ContainerStarted","Data":"cbc00837ecd4f45905d8d6b06bc50ea9d58636b406bc94c1fa686b6c1dce6211"} Sep 17 15:36:05.207215 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:05.207186 2581 generic.go:356] "Generic (PLEG): container finished" podID="e09031a0-e098-485a-ac1e-99a82fffc196" containerID="926b0d5dc0e93df421db715558a2b6e8bf13c08226a76d1748e67b657eb4d376" exitCode=0 Sep 17 15:36:05.207307 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:05.207230 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" event={"ID":"e09031a0-e098-485a-ac1e-99a82fffc196","Type":"ContainerDied","Data":"926b0d5dc0e93df421db715558a2b6e8bf13c08226a76d1748e67b657eb4d376"} Sep 17 15:36:05.207307 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:05.207269 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" event={"ID":"e09031a0-e098-485a-ac1e-99a82fffc196","Type":"ContainerStarted","Data":"60a18471449f61fad8babc0451b924e95d7354207495c6b7c7206090de8641a7"} Sep 17 15:36:05.208512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:05.208490 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 15:36:05.256682 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:05.256652 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks"] Sep 17 15:36:05.294613 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:36:05.294587 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881dc300_89e8_4a8f_a39a_2402449dd192.slice/crio-f256e69ae47ce68d2db319ce0646f8fa3873941e462c15e6cd69a43ab927f633 WatchSource:0}: Error finding container f256e69ae47ce68d2db319ce0646f8fa3873941e462c15e6cd69a43ab927f633: Status 404 returned error can't find the container with id f256e69ae47ce68d2db319ce0646f8fa3873941e462c15e6cd69a43ab927f633 Sep 17 15:36:06.213578 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:06.213510 2581 generic.go:356] "Generic (PLEG): container finished" podID="f4efa66a-d1af-479f-9972-e513d718ecbe" containerID="2cc1be690841d8f2402dc04c6e8ac97742c3763e678f1d50347d7a8eb4b55b4c" exitCode=0 Sep 17 15:36:06.213976 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:06.213597 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" event={"ID":"f4efa66a-d1af-479f-9972-e513d718ecbe","Type":"ContainerDied","Data":"2cc1be690841d8f2402dc04c6e8ac97742c3763e678f1d50347d7a8eb4b55b4c"} Sep 17 15:36:06.215258 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:06.215237 2581 generic.go:356] "Generic (PLEG): container finished" podID="881dc300-89e8-4a8f-a39a-2402449dd192" containerID="a784710c35f23e366daa1179911ef7bf54863ce93e62a47c4b05fbf79987bb67" exitCode=0 Sep 17 15:36:06.215355 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:06.215320 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" event={"ID":"881dc300-89e8-4a8f-a39a-2402449dd192","Type":"ContainerDied","Data":"a784710c35f23e366daa1179911ef7bf54863ce93e62a47c4b05fbf79987bb67"} Sep 17 15:36:06.215429 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:06.215393 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" event={"ID":"881dc300-89e8-4a8f-a39a-2402449dd192","Type":"ContainerStarted","Data":"f256e69ae47ce68d2db319ce0646f8fa3873941e462c15e6cd69a43ab927f633"} Sep 17 15:36:06.217090 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:06.217065 2581 generic.go:356] "Generic (PLEG): container finished" podID="0a9a76a5-c8aa-45d1-a559-f82a39a2f305" containerID="752fc570271eb09adef63dd8dfe40e4c634be5977f21beaa4b0b21de515dc374" exitCode=0 Sep 17 15:36:06.217160 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:06.217114 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" event={"ID":"0a9a76a5-c8aa-45d1-a559-f82a39a2f305","Type":"ContainerDied","Data":"752fc570271eb09adef63dd8dfe40e4c634be5977f21beaa4b0b21de515dc374"} Sep 17 15:36:07.223712 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:07.223681 2581 generic.go:356] "Generic (PLEG): container finished" podID="f4efa66a-d1af-479f-9972-e513d718ecbe" containerID="585355adec0443b217821a38e25cf9738004b0fcf7b38749475f723a5039d71f" exitCode=0 Sep 17 15:36:07.224085 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:07.223764 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" event={"ID":"f4efa66a-d1af-479f-9972-e513d718ecbe","Type":"ContainerDied","Data":"585355adec0443b217821a38e25cf9738004b0fcf7b38749475f723a5039d71f"} Sep 17 15:36:07.225776 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:07.225754 2581 generic.go:356] "Generic (PLEG): container finished" podID="e09031a0-e098-485a-ac1e-99a82fffc196" containerID="d9b6bc6a4423eb98ddbed43ca1094777a88fc89d74a7f970d82305b30bb0a94c" exitCode=0 Sep 17 15:36:07.225993 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:07.225820 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" event={"ID":"e09031a0-e098-485a-ac1e-99a82fffc196","Type":"ContainerDied","Data":"d9b6bc6a4423eb98ddbed43ca1094777a88fc89d74a7f970d82305b30bb0a94c"} Sep 17 15:36:07.643039 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:07.642996 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:07.733464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:07.733435 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:08.232714 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:08.232679 2581 generic.go:356] "Generic (PLEG): container finished" podID="e09031a0-e098-485a-ac1e-99a82fffc196" containerID="35ac3220a362e6924e431ddd92a6e418c9584c617f6dea52946692a18521270a" exitCode=0 Sep 17 15:36:08.233119 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:08.232770 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" event={"ID":"e09031a0-e098-485a-ac1e-99a82fffc196","Type":"ContainerDied","Data":"35ac3220a362e6924e431ddd92a6e418c9584c617f6dea52946692a18521270a"} Sep 17 15:36:08.234572 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:08.234551 2581 generic.go:356] "Generic (PLEG): container finished" podID="881dc300-89e8-4a8f-a39a-2402449dd192" containerID="0c2c54e59bbead21ebc49928f3b267dd25a1a6a594cb6be7f0786072d91a5535" exitCode=0 Sep 17 15:36:08.234670 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:08.234623 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" event={"ID":"881dc300-89e8-4a8f-a39a-2402449dd192","Type":"ContainerDied","Data":"0c2c54e59bbead21ebc49928f3b267dd25a1a6a594cb6be7f0786072d91a5535"} Sep 17 15:36:08.236865 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:08.236840 2581 generic.go:356] "Generic (PLEG): container finished" podID="f4efa66a-d1af-479f-9972-e513d718ecbe" containerID="8ce23ff8d176957c5d6ff675c345f1b1d101e07fbb2ae78e015a5337fe8c3f88" exitCode=0 Sep 17 15:36:08.236976 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:08.236894 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" event={"ID":"f4efa66a-d1af-479f-9972-e513d718ecbe","Type":"ContainerDied","Data":"8ce23ff8d176957c5d6ff675c345f1b1d101e07fbb2ae78e015a5337fe8c3f88"} Sep 17 15:36:09.244532 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.244494 2581 generic.go:356] "Generic (PLEG): container finished" podID="881dc300-89e8-4a8f-a39a-2402449dd192" containerID="8532e2315701cf596553207a12c9d2930eb39f9c85888b64944049792655ee92" exitCode=0 Sep 17 15:36:09.244903 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.244571 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" event={"ID":"881dc300-89e8-4a8f-a39a-2402449dd192","Type":"ContainerDied","Data":"8532e2315701cf596553207a12c9d2930eb39f9c85888b64944049792655ee92"} Sep 17 15:36:09.368680 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.368656 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" Sep 17 15:36:09.394943 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.394905 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" Sep 17 15:36:09.457690 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.457658 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz4bn\" (UniqueName: \"kubernetes.io/projected/f4efa66a-d1af-479f-9972-e513d718ecbe-kube-api-access-rz4bn\") pod \"f4efa66a-d1af-479f-9972-e513d718ecbe\" (UID: \"f4efa66a-d1af-479f-9972-e513d718ecbe\") " Sep 17 15:36:09.457856 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.457709 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8754g\" (UniqueName: \"kubernetes.io/projected/e09031a0-e098-485a-ac1e-99a82fffc196-kube-api-access-8754g\") pod \"e09031a0-e098-485a-ac1e-99a82fffc196\" (UID: \"e09031a0-e098-485a-ac1e-99a82fffc196\") " Sep 17 15:36:09.457856 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.457729 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4efa66a-d1af-479f-9972-e513d718ecbe-util\") pod \"f4efa66a-d1af-479f-9972-e513d718ecbe\" (UID: \"f4efa66a-d1af-479f-9972-e513d718ecbe\") " Sep 17 15:36:09.457856 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.457766 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e09031a0-e098-485a-ac1e-99a82fffc196-util\") pod \"e09031a0-e098-485a-ac1e-99a82fffc196\" (UID: \"e09031a0-e098-485a-ac1e-99a82fffc196\") " Sep 17 15:36:09.457856 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.457818 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e09031a0-e098-485a-ac1e-99a82fffc196-bundle\") pod \"e09031a0-e098-485a-ac1e-99a82fffc196\" (UID: \"e09031a0-e098-485a-ac1e-99a82fffc196\") " Sep 17 15:36:09.458093 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.457877 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4efa66a-d1af-479f-9972-e513d718ecbe-bundle\") pod \"f4efa66a-d1af-479f-9972-e513d718ecbe\" (UID: \"f4efa66a-d1af-479f-9972-e513d718ecbe\") " Sep 17 15:36:09.458774 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.458502 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09031a0-e098-485a-ac1e-99a82fffc196-bundle" (OuterVolumeSpecName: "bundle") pod "e09031a0-e098-485a-ac1e-99a82fffc196" (UID: "e09031a0-e098-485a-ac1e-99a82fffc196"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:36:09.458895 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.458820 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4efa66a-d1af-479f-9972-e513d718ecbe-bundle" (OuterVolumeSpecName: "bundle") pod "f4efa66a-d1af-479f-9972-e513d718ecbe" (UID: "f4efa66a-d1af-479f-9972-e513d718ecbe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:36:09.460067 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.460037 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4efa66a-d1af-479f-9972-e513d718ecbe-kube-api-access-rz4bn" (OuterVolumeSpecName: "kube-api-access-rz4bn") pod "f4efa66a-d1af-479f-9972-e513d718ecbe" (UID: "f4efa66a-d1af-479f-9972-e513d718ecbe"). InnerVolumeSpecName "kube-api-access-rz4bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:36:09.460185 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.460073 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09031a0-e098-485a-ac1e-99a82fffc196-kube-api-access-8754g" (OuterVolumeSpecName: "kube-api-access-8754g") pod "e09031a0-e098-485a-ac1e-99a82fffc196" (UID: "e09031a0-e098-485a-ac1e-99a82fffc196"). InnerVolumeSpecName "kube-api-access-8754g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:36:09.466307 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.466276 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4efa66a-d1af-479f-9972-e513d718ecbe-util" (OuterVolumeSpecName: "util") pod "f4efa66a-d1af-479f-9972-e513d718ecbe" (UID: "f4efa66a-d1af-479f-9972-e513d718ecbe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:36:09.466830 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.466803 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09031a0-e098-485a-ac1e-99a82fffc196-util" (OuterVolumeSpecName: "util") pod "e09031a0-e098-485a-ac1e-99a82fffc196" (UID: "e09031a0-e098-485a-ac1e-99a82fffc196"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:36:09.559240 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.559209 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8754g\" (UniqueName: \"kubernetes.io/projected/e09031a0-e098-485a-ac1e-99a82fffc196-kube-api-access-8754g\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:09.559240 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.559238 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4efa66a-d1af-479f-9972-e513d718ecbe-util\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:09.559408 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.559248 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e09031a0-e098-485a-ac1e-99a82fffc196-util\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:09.559408 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.559256 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e09031a0-e098-485a-ac1e-99a82fffc196-bundle\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:09.559408 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.559263 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4efa66a-d1af-479f-9972-e513d718ecbe-bundle\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:09.559408 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:09.559271 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz4bn\" (UniqueName: \"kubernetes.io/projected/f4efa66a-d1af-479f-9972-e513d718ecbe-kube-api-access-rz4bn\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:10.251148 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.251122 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" Sep 17 15:36:10.251618 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.251124 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ggzs" event={"ID":"f4efa66a-d1af-479f-9972-e513d718ecbe","Type":"ContainerDied","Data":"cbc00837ecd4f45905d8d6b06bc50ea9d58636b406bc94c1fa686b6c1dce6211"} Sep 17 15:36:10.251618 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.251238 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbc00837ecd4f45905d8d6b06bc50ea9d58636b406bc94c1fa686b6c1dce6211" Sep 17 15:36:10.253089 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.253065 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" event={"ID":"e09031a0-e098-485a-ac1e-99a82fffc196","Type":"ContainerDied","Data":"60a18471449f61fad8babc0451b924e95d7354207495c6b7c7206090de8641a7"} Sep 17 15:36:10.253089 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.253087 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmfn2b" Sep 17 15:36:10.253258 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.253101 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60a18471449f61fad8babc0451b924e95d7354207495c6b7c7206090de8641a7" Sep 17 15:36:10.380529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.380507 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" Sep 17 15:36:10.467243 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.467217 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/881dc300-89e8-4a8f-a39a-2402449dd192-bundle\") pod \"881dc300-89e8-4a8f-a39a-2402449dd192\" (UID: \"881dc300-89e8-4a8f-a39a-2402449dd192\") " Sep 17 15:36:10.467374 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.467267 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzqbl\" (UniqueName: \"kubernetes.io/projected/881dc300-89e8-4a8f-a39a-2402449dd192-kube-api-access-gzqbl\") pod \"881dc300-89e8-4a8f-a39a-2402449dd192\" (UID: \"881dc300-89e8-4a8f-a39a-2402449dd192\") " Sep 17 15:36:10.467374 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.467324 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/881dc300-89e8-4a8f-a39a-2402449dd192-util\") pod \"881dc300-89e8-4a8f-a39a-2402449dd192\" (UID: \"881dc300-89e8-4a8f-a39a-2402449dd192\") " Sep 17 15:36:10.467742 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.467715 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881dc300-89e8-4a8f-a39a-2402449dd192-bundle" (OuterVolumeSpecName: "bundle") pod "881dc300-89e8-4a8f-a39a-2402449dd192" (UID: "881dc300-89e8-4a8f-a39a-2402449dd192"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:36:10.469371 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.469340 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881dc300-89e8-4a8f-a39a-2402449dd192-kube-api-access-gzqbl" (OuterVolumeSpecName: "kube-api-access-gzqbl") pod "881dc300-89e8-4a8f-a39a-2402449dd192" (UID: "881dc300-89e8-4a8f-a39a-2402449dd192"). InnerVolumeSpecName "kube-api-access-gzqbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:36:10.472537 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.472515 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881dc300-89e8-4a8f-a39a-2402449dd192-util" (OuterVolumeSpecName: "util") pod "881dc300-89e8-4a8f-a39a-2402449dd192" (UID: "881dc300-89e8-4a8f-a39a-2402449dd192"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:36:10.568722 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.568698 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/881dc300-89e8-4a8f-a39a-2402449dd192-util\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:10.568722 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.568721 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/881dc300-89e8-4a8f-a39a-2402449dd192-bundle\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:10.568869 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:10.568731 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzqbl\" (UniqueName: \"kubernetes.io/projected/881dc300-89e8-4a8f-a39a-2402449dd192-kube-api-access-gzqbl\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:11.259802 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:11.259773 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" Sep 17 15:36:11.259802 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:11.259795 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7f5cks" event={"ID":"881dc300-89e8-4a8f-a39a-2402449dd192","Type":"ContainerDied","Data":"f256e69ae47ce68d2db319ce0646f8fa3873941e462c15e6cd69a43ab927f633"} Sep 17 15:36:11.260353 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:11.259836 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f256e69ae47ce68d2db319ce0646f8fa3873941e462c15e6cd69a43ab927f633" Sep 17 15:36:13.271246 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:13.271169 2581 generic.go:356] "Generic (PLEG): container finished" podID="0a9a76a5-c8aa-45d1-a559-f82a39a2f305" containerID="83f2d9b93976b655a27ccd4701b86cde7b2fbbef70219eb63131dfa2d151e5ba" exitCode=0 Sep 17 15:36:13.271246 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:13.271231 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" event={"ID":"0a9a76a5-c8aa-45d1-a559-f82a39a2f305","Type":"ContainerDied","Data":"83f2d9b93976b655a27ccd4701b86cde7b2fbbef70219eb63131dfa2d151e5ba"} Sep 17 15:36:14.277975 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:14.277937 2581 generic.go:356] "Generic (PLEG): container finished" podID="0a9a76a5-c8aa-45d1-a559-f82a39a2f305" containerID="eef3594426526e78d23042950dbc1957ad1df3c44b0c62c3564ef483528b3442" exitCode=0 Sep 17 15:36:14.278397 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:14.278002 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" event={"ID":"0a9a76a5-c8aa-45d1-a559-f82a39a2f305","Type":"ContainerDied","Data":"eef3594426526e78d23042950dbc1957ad1df3c44b0c62c3564ef483528b3442"} Sep 17 15:36:15.407462 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:15.407437 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" Sep 17 15:36:15.505392 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:15.505361 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-bundle\") pod \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\" (UID: \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\") " Sep 17 15:36:15.505392 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:15.505401 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbw7m\" (UniqueName: \"kubernetes.io/projected/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-kube-api-access-sbw7m\") pod \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\" (UID: \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\") " Sep 17 15:36:15.505580 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:15.505429 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-util\") pod \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\" (UID: \"0a9a76a5-c8aa-45d1-a559-f82a39a2f305\") " Sep 17 15:36:15.505885 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:15.505860 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-bundle" (OuterVolumeSpecName: "bundle") pod "0a9a76a5-c8aa-45d1-a559-f82a39a2f305" (UID: "0a9a76a5-c8aa-45d1-a559-f82a39a2f305"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:36:15.507450 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:15.507434 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-kube-api-access-sbw7m" (OuterVolumeSpecName: "kube-api-access-sbw7m") pod "0a9a76a5-c8aa-45d1-a559-f82a39a2f305" (UID: "0a9a76a5-c8aa-45d1-a559-f82a39a2f305"). InnerVolumeSpecName "kube-api-access-sbw7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:36:15.511742 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:15.511701 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-util" (OuterVolumeSpecName: "util") pod "0a9a76a5-c8aa-45d1-a559-f82a39a2f305" (UID: "0a9a76a5-c8aa-45d1-a559-f82a39a2f305"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:36:15.606393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:15.606329 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbw7m\" (UniqueName: \"kubernetes.io/projected/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-kube-api-access-sbw7m\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:15.606393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:15.606358 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-util\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:15.606393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:15.606371 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a9a76a5-c8aa-45d1-a559-f82a39a2f305-bundle\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:16.288796 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:16.288765 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" Sep 17 15:36:16.288976 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:16.288791 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6g9pph" event={"ID":"0a9a76a5-c8aa-45d1-a559-f82a39a2f305","Type":"ContainerDied","Data":"8f8e230f5d314c4222cec460e6dd9a005985bea4bb9dc9e379d29e19819c8b7a"} Sep 17 15:36:16.288976 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:16.288863 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f8e230f5d314c4222cec460e6dd9a005985bea4bb9dc9e379d29e19819c8b7a" Sep 17 15:36:18.204925 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:18.204881 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-594df7bf64-p25tc"] Sep 17 15:36:18.372082 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:18.372057 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:18.596902 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:18.596873 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:22.569242 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:22.569210 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:25.143516 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:25.143484 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:27.011335 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:27.011297 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:27.924044 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:27.924003 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:28.123821 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:28.123791 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:28.443176 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:28.443141 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:28.779283 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:28.779258 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:28.929679 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:28.929639 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:32.416749 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:32.416719 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:32.849449 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:32.849422 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:34.588842 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:34.588813 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:34.826619 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:34.826591 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:35.714832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.714796 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-2ttvj"] Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715103 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e09031a0-e098-485a-ac1e-99a82fffc196" containerName="util" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715116 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e09031a0-e098-485a-ac1e-99a82fffc196" containerName="util" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715129 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4efa66a-d1af-479f-9972-e513d718ecbe" containerName="extract" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715135 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4efa66a-d1af-479f-9972-e513d718ecbe" containerName="extract" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715142 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9a76a5-c8aa-45d1-a559-f82a39a2f305" containerName="pull" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715147 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9a76a5-c8aa-45d1-a559-f82a39a2f305" containerName="pull" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715155 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4efa66a-d1af-479f-9972-e513d718ecbe" containerName="util" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715160 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4efa66a-d1af-479f-9972-e513d718ecbe" containerName="util" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715166 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9a76a5-c8aa-45d1-a559-f82a39a2f305" containerName="extract" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715171 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9a76a5-c8aa-45d1-a559-f82a39a2f305" containerName="extract" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715178 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="881dc300-89e8-4a8f-a39a-2402449dd192" containerName="util" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715184 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="881dc300-89e8-4a8f-a39a-2402449dd192" containerName="util" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715189 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="881dc300-89e8-4a8f-a39a-2402449dd192" containerName="pull" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715194 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="881dc300-89e8-4a8f-a39a-2402449dd192" containerName="pull" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715200 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9a76a5-c8aa-45d1-a559-f82a39a2f305" containerName="util" Sep 17 15:36:35.715200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715205 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9a76a5-c8aa-45d1-a559-f82a39a2f305" containerName="util" Sep 17 15:36:35.715674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715211 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4efa66a-d1af-479f-9972-e513d718ecbe" containerName="pull" Sep 17 15:36:35.715674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715216 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4efa66a-d1af-479f-9972-e513d718ecbe" containerName="pull" Sep 17 15:36:35.715674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715222 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e09031a0-e098-485a-ac1e-99a82fffc196" containerName="extract" Sep 17 15:36:35.715674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715226 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e09031a0-e098-485a-ac1e-99a82fffc196" containerName="extract" Sep 17 15:36:35.715674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715231 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="881dc300-89e8-4a8f-a39a-2402449dd192" containerName="extract" Sep 17 15:36:35.715674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715236 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="881dc300-89e8-4a8f-a39a-2402449dd192" containerName="extract" Sep 17 15:36:35.715674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715243 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e09031a0-e098-485a-ac1e-99a82fffc196" containerName="pull" Sep 17 15:36:35.715674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715247 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e09031a0-e098-485a-ac1e-99a82fffc196" containerName="pull" Sep 17 15:36:35.715674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715291 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4efa66a-d1af-479f-9972-e513d718ecbe" containerName="extract" Sep 17 15:36:35.715674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715298 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e09031a0-e098-485a-ac1e-99a82fffc196" containerName="extract" Sep 17 15:36:35.715674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715304 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="881dc300-89e8-4a8f-a39a-2402449dd192" containerName="extract" Sep 17 15:36:35.715674 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.715309 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9a76a5-c8aa-45d1-a559-f82a39a2f305" containerName="extract" Sep 17 15:36:35.719393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.719377 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-2ttvj" Sep 17 15:36:35.722756 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.722731 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 17 15:36:35.723210 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.723189 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 17 15:36:35.723683 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.723667 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-v2pcq\"" Sep 17 15:36:35.740099 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.740077 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-2ttvj"] Sep 17 15:36:35.865053 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.865018 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5lmn\" (UniqueName: \"kubernetes.io/projected/54d0c38e-a5d1-4413-a14e-738e86f064ba-kube-api-access-g5lmn\") pod \"authorino-operator-76b478bd78-2ttvj\" (UID: \"54d0c38e-a5d1-4413-a14e-738e86f064ba\") " pod="kuadrant-system/authorino-operator-76b478bd78-2ttvj" Sep 17 15:36:35.966393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.966316 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5lmn\" (UniqueName: \"kubernetes.io/projected/54d0c38e-a5d1-4413-a14e-738e86f064ba-kube-api-access-g5lmn\") pod \"authorino-operator-76b478bd78-2ttvj\" (UID: \"54d0c38e-a5d1-4413-a14e-738e86f064ba\") " pod="kuadrant-system/authorino-operator-76b478bd78-2ttvj" Sep 17 15:36:35.992760 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:35.992738 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5lmn\" (UniqueName: \"kubernetes.io/projected/54d0c38e-a5d1-4413-a14e-738e86f064ba-kube-api-access-g5lmn\") pod \"authorino-operator-76b478bd78-2ttvj\" (UID: \"54d0c38e-a5d1-4413-a14e-738e86f064ba\") " pod="kuadrant-system/authorino-operator-76b478bd78-2ttvj" Sep 17 15:36:36.029180 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:36.029153 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-2ttvj" Sep 17 15:36:36.340349 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:36.340324 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-2ttvj"] Sep 17 15:36:36.342382 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:36:36.342348 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d0c38e_a5d1_4413_a14e_738e86f064ba.slice/crio-e690c58dbb5aeac362a30e01f36bffb09f3c1259aa3c1e62c79f1d968a245311 WatchSource:0}: Error finding container e690c58dbb5aeac362a30e01f36bffb09f3c1259aa3c1e62c79f1d968a245311: Status 404 returned error can't find the container with id e690c58dbb5aeac362a30e01f36bffb09f3c1259aa3c1e62c79f1d968a245311 Sep 17 15:36:36.386378 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:36.386346 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-2ttvj" event={"ID":"54d0c38e-a5d1-4413-a14e-738e86f064ba","Type":"ContainerStarted","Data":"e690c58dbb5aeac362a30e01f36bffb09f3c1259aa3c1e62c79f1d968a245311"} Sep 17 15:36:36.728076 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:36.727997 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:37.092106 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:37.092077 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:39.406739 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:39.406701 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-2ttvj" event={"ID":"54d0c38e-a5d1-4413-a14e-738e86f064ba","Type":"ContainerStarted","Data":"ec112b589c7fc12d1541d65505f6b38947589e6578c38501b68c2f217bfbaf49"} Sep 17 15:36:39.407138 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:39.406840 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-2ttvj" Sep 17 15:36:40.168101 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.168068 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:40.734682 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.734624 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-2ttvj" podStartSLOduration=3.251173273 podStartE2EDuration="5.73460361s" totalImagesPullingTime="2.483430337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:36:39.48110559 +0000 UTC m=+386.477342335" watchObservedRunningTime="2026-09-17 15:36:40.73460361 +0000 UTC m=+387.730840358" Sep 17 15:36:40.735098 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.734809 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb"] Sep 17 15:36:40.739392 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.739368 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" Sep 17 15:36:40.774670 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.774632 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb"] Sep 17 15:36:40.779132 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.779099 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4cmh6\"" Sep 17 15:36:40.779353 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.779339 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 17 15:36:40.779491 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.779479 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 17 15:36:40.803633 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.803602 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/899f6280-8bbb-41ca-86a7-9cb7f95ada73-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-kknfb\" (UID: \"899f6280-8bbb-41ca-86a7-9cb7f95ada73\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" Sep 17 15:36:40.803764 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.803657 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/899f6280-8bbb-41ca-86a7-9cb7f95ada73-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kknfb\" (UID: \"899f6280-8bbb-41ca-86a7-9cb7f95ada73\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" Sep 17 15:36:40.803764 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.803734 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f9r6\" (UniqueName: \"kubernetes.io/projected/899f6280-8bbb-41ca-86a7-9cb7f95ada73-kube-api-access-6f9r6\") pod \"kuadrant-console-plugin-fbdd4f445-kknfb\" (UID: \"899f6280-8bbb-41ca-86a7-9cb7f95ada73\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" Sep 17 15:36:40.806561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.806540 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:40.904215 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.904182 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/899f6280-8bbb-41ca-86a7-9cb7f95ada73-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-kknfb\" (UID: \"899f6280-8bbb-41ca-86a7-9cb7f95ada73\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" Sep 17 15:36:40.904215 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.904221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/899f6280-8bbb-41ca-86a7-9cb7f95ada73-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kknfb\" (UID: \"899f6280-8bbb-41ca-86a7-9cb7f95ada73\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" Sep 17 15:36:40.904480 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.904274 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f9r6\" (UniqueName: \"kubernetes.io/projected/899f6280-8bbb-41ca-86a7-9cb7f95ada73-kube-api-access-6f9r6\") pod \"kuadrant-console-plugin-fbdd4f445-kknfb\" (UID: \"899f6280-8bbb-41ca-86a7-9cb7f95ada73\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" Sep 17 15:36:40.904866 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.904843 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/899f6280-8bbb-41ca-86a7-9cb7f95ada73-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-kknfb\" (UID: \"899f6280-8bbb-41ca-86a7-9cb7f95ada73\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" Sep 17 15:36:40.906604 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.906587 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/899f6280-8bbb-41ca-86a7-9cb7f95ada73-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-kknfb\" (UID: \"899f6280-8bbb-41ca-86a7-9cb7f95ada73\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" Sep 17 15:36:40.933022 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:40.932998 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f9r6\" (UniqueName: \"kubernetes.io/projected/899f6280-8bbb-41ca-86a7-9cb7f95ada73-kube-api-access-6f9r6\") pod \"kuadrant-console-plugin-fbdd4f445-kknfb\" (UID: \"899f6280-8bbb-41ca-86a7-9cb7f95ada73\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" Sep 17 15:36:41.026101 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:41.026068 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:41.048093 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:41.048069 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" Sep 17 15:36:41.365982 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:41.365957 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb"] Sep 17 15:36:41.368410 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:36:41.368384 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod899f6280_8bbb_41ca_86a7_9cb7f95ada73.slice/crio-5e8de3de1b50b04afde931547c4e85d695de89b306061535ce67f5c9f544f850 WatchSource:0}: Error finding container 5e8de3de1b50b04afde931547c4e85d695de89b306061535ce67f5c9f544f850: Status 404 returned error can't find the container with id 5e8de3de1b50b04afde931547c4e85d695de89b306061535ce67f5c9f544f850 Sep 17 15:36:41.417434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:41.417403 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" event={"ID":"899f6280-8bbb-41ca-86a7-9cb7f95ada73","Type":"ContainerStarted","Data":"5e8de3de1b50b04afde931547c4e85d695de89b306061535ce67f5c9f544f850"} Sep 17 15:36:41.900892 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:41.900860 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:43.223724 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.223679 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-594df7bf64-p25tc" podUID="e191da55-d622-4b30-801d-c537e6b18270" containerName="console" containerID="cri-o://4af340571cfae933af9728bfd1e2aa7fc8e5e6c3c2b5fd5237d130ad3ac68560" gracePeriod=15 Sep 17 15:36:43.432309 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.432272 2581 generic.go:356] "Generic (PLEG): container finished" podID="e191da55-d622-4b30-801d-c537e6b18270" containerID="4af340571cfae933af9728bfd1e2aa7fc8e5e6c3c2b5fd5237d130ad3ac68560" exitCode=0 Sep 17 15:36:43.432448 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.432352 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-594df7bf64-p25tc" event={"ID":"e191da55-d622-4b30-801d-c537e6b18270","Type":"ContainerDied","Data":"4af340571cfae933af9728bfd1e2aa7fc8e5e6c3c2b5fd5237d130ad3ac68560"} Sep 17 15:36:43.487598 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.487528 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:36:43.526086 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.526051 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpnkx\" (UniqueName: \"kubernetes.io/projected/e191da55-d622-4b30-801d-c537e6b18270-kube-api-access-bpnkx\") pod \"e191da55-d622-4b30-801d-c537e6b18270\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " Sep 17 15:36:43.526086 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.526089 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-console-config\") pod \"e191da55-d622-4b30-801d-c537e6b18270\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " Sep 17 15:36:43.526310 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.526114 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e191da55-d622-4b30-801d-c537e6b18270-console-oauth-config\") pod \"e191da55-d622-4b30-801d-c537e6b18270\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " Sep 17 15:36:43.526310 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.526154 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e191da55-d622-4b30-801d-c537e6b18270-console-serving-cert\") pod \"e191da55-d622-4b30-801d-c537e6b18270\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " Sep 17 15:36:43.526310 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.526265 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-oauth-serving-cert\") pod \"e191da55-d622-4b30-801d-c537e6b18270\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " Sep 17 15:36:43.526464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.526322 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-trusted-ca-bundle\") pod \"e191da55-d622-4b30-801d-c537e6b18270\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " Sep 17 15:36:43.526464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.526353 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-service-ca\") pod \"e191da55-d622-4b30-801d-c537e6b18270\" (UID: \"e191da55-d622-4b30-801d-c537e6b18270\") " Sep 17 15:36:43.526599 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.526575 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-console-config" (OuterVolumeSpecName: "console-config") pod "e191da55-d622-4b30-801d-c537e6b18270" (UID: "e191da55-d622-4b30-801d-c537e6b18270"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:36:43.526744 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.526639 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e191da55-d622-4b30-801d-c537e6b18270" (UID: "e191da55-d622-4b30-801d-c537e6b18270"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:36:43.526837 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.526814 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e191da55-d622-4b30-801d-c537e6b18270" (UID: "e191da55-d622-4b30-801d-c537e6b18270"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:36:43.526905 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.526879 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-service-ca" (OuterVolumeSpecName: "service-ca") pod "e191da55-d622-4b30-801d-c537e6b18270" (UID: "e191da55-d622-4b30-801d-c537e6b18270"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:36:43.528546 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.528522 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e191da55-d622-4b30-801d-c537e6b18270-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e191da55-d622-4b30-801d-c537e6b18270" (UID: "e191da55-d622-4b30-801d-c537e6b18270"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:36:43.528546 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.528532 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e191da55-d622-4b30-801d-c537e6b18270-kube-api-access-bpnkx" (OuterVolumeSpecName: "kube-api-access-bpnkx") pod "e191da55-d622-4b30-801d-c537e6b18270" (UID: "e191da55-d622-4b30-801d-c537e6b18270"). InnerVolumeSpecName "kube-api-access-bpnkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:36:43.528701 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.528648 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e191da55-d622-4b30-801d-c537e6b18270-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e191da55-d622-4b30-801d-c537e6b18270" (UID: "e191da55-d622-4b30-801d-c537e6b18270"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:36:43.627591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.627550 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-trusted-ca-bundle\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:43.627591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.627585 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:43.627591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.627595 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpnkx\" (UniqueName: \"kubernetes.io/projected/e191da55-d622-4b30-801d-c537e6b18270-kube-api-access-bpnkx\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:43.627814 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.627604 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-console-config\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:43.627814 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.627613 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e191da55-d622-4b30-801d-c537e6b18270-console-oauth-config\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:43.627814 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.627621 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e191da55-d622-4b30-801d-c537e6b18270-console-serving-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:43.627814 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:43.627633 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e191da55-d622-4b30-801d-c537e6b18270-oauth-serving-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:36:44.440234 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:44.440203 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-594df7bf64-p25tc" Sep 17 15:36:44.440727 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:44.440202 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-594df7bf64-p25tc" event={"ID":"e191da55-d622-4b30-801d-c537e6b18270","Type":"ContainerDied","Data":"a56f4a46db0cbc97a8b9704577052812c344a6387e96ffbfb0592bacaca5fe7c"} Sep 17 15:36:44.440727 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:44.440365 2581 scope.go:117] "RemoveContainer" containerID="4af340571cfae933af9728bfd1e2aa7fc8e5e6c3c2b5fd5237d130ad3ac68560" Sep 17 15:36:44.478519 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:44.478488 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-594df7bf64-p25tc"] Sep 17 15:36:44.509266 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:44.509235 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-594df7bf64-p25tc"] Sep 17 15:36:44.912299 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:44.912252 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:45.151979 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:45.151946 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:45.462581 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:45.462548 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e191da55-d622-4b30-801d-c537e6b18270" path="/var/lib/kubelet/pods/e191da55-d622-4b30-801d-c537e6b18270/volumes" Sep 17 15:36:45.529091 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:45.529060 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:45.881566 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:45.881471 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:46.250869 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:46.250845 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:46.451476 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:46.451373 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" event={"ID":"899f6280-8bbb-41ca-86a7-9cb7f95ada73","Type":"ContainerStarted","Data":"92fa3508680ef6209e6ab84b601d371e29c53208ede86471e2fb58d4048f4aaf"} Sep 17 15:36:46.490530 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:46.488432 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-kknfb" podStartSLOduration=1.697170994 podStartE2EDuration="6.488408869s" totalImagesPullingTime="4.791237875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:36:46.48543164 +0000 UTC m=+393.481668388" watchObservedRunningTime="2026-09-17 15:36:46.488408869 +0000 UTC m=+393.484645616" Sep 17 15:36:50.414804 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:50.414772 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-2ttvj" Sep 17 15:36:53.097242 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:53.097210 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:54.556248 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:54.556217 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:54.815417 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:54.815311 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:55.053514 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:55.053481 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:57.165029 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:57.164996 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:36:57.546246 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:36:57.546216 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:00.901949 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:00.901901 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:01.145646 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:01.145614 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:05.502900 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:05.502862 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:05.813697 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:05.813660 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:06.472861 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:06.472824 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:06.706091 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:06.706064 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:06.991137 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:06.991108 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:07.437046 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:07.437010 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:14.817398 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:14.817369 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:15.027315 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:15.027278 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:15.188832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:15.188752 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:30.355001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:30.354965 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:30.560761 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:30.560718 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:32.678149 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:32.678115 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:32.772431 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:32.772385 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:32.885693 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:32.885657 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:33.589782 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.589749 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-r7kjt"] Sep 17 15:37:33.590102 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.590087 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e191da55-d622-4b30-801d-c537e6b18270" containerName="console" Sep 17 15:37:33.590102 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.590102 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e191da55-d622-4b30-801d-c537e6b18270" containerName="console" Sep 17 15:37:33.590265 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.590172 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e191da55-d622-4b30-801d-c537e6b18270" containerName="console" Sep 17 15:37:33.607085 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.607062 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-r7kjt" Sep 17 15:37:33.609821 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.609791 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-c54t2\"" Sep 17 15:37:33.612261 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.612237 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-r7kjt"] Sep 17 15:37:33.691340 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.691298 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:33.739899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.739875 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48mlr\" (UniqueName: \"kubernetes.io/projected/c91d0865-e184-4989-8bde-d349073aefc3-kube-api-access-48mlr\") pod \"authorino-7bcfd8f9cd-r7kjt\" (UID: \"c91d0865-e184-4989-8bde-d349073aefc3\") " pod="kuadrant-system/authorino-7bcfd8f9cd-r7kjt" Sep 17 15:37:33.840588 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.840520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48mlr\" (UniqueName: \"kubernetes.io/projected/c91d0865-e184-4989-8bde-d349073aefc3-kube-api-access-48mlr\") pod \"authorino-7bcfd8f9cd-r7kjt\" (UID: \"c91d0865-e184-4989-8bde-d349073aefc3\") " pod="kuadrant-system/authorino-7bcfd8f9cd-r7kjt" Sep 17 15:37:33.870329 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.870298 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48mlr\" (UniqueName: \"kubernetes.io/projected/c91d0865-e184-4989-8bde-d349073aefc3-kube-api-access-48mlr\") pod \"authorino-7bcfd8f9cd-r7kjt\" (UID: \"c91d0865-e184-4989-8bde-d349073aefc3\") " pod="kuadrant-system/authorino-7bcfd8f9cd-r7kjt" Sep 17 15:37:33.904833 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.904808 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-nw49b"] Sep 17 15:37:33.907268 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.907249 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nw49b" Sep 17 15:37:33.916506 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.916482 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-r7kjt" Sep 17 15:37:33.918942 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.918572 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:33.919357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:33.919334 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nw49b"] Sep 17 15:37:34.042465 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:34.042432 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjjrq\" (UniqueName: \"kubernetes.io/projected/aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e-kube-api-access-cjjrq\") pod \"authorino-59c974bd48-nw49b\" (UID: \"aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e\") " pod="kuadrant-system/authorino-59c974bd48-nw49b" Sep 17 15:37:34.143868 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:34.143796 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjjrq\" (UniqueName: \"kubernetes.io/projected/aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e-kube-api-access-cjjrq\") pod \"authorino-59c974bd48-nw49b\" (UID: \"aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e\") " pod="kuadrant-system/authorino-59c974bd48-nw49b" Sep 17 15:37:34.153984 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:34.153956 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjjrq\" (UniqueName: \"kubernetes.io/projected/aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e-kube-api-access-cjjrq\") pod \"authorino-59c974bd48-nw49b\" (UID: \"aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e\") " pod="kuadrant-system/authorino-59c974bd48-nw49b" Sep 17 15:37:34.194398 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:34.194378 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-r7kjt"] Sep 17 15:37:34.196852 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:37:34.196828 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc91d0865_e184_4989_8bde_d349073aefc3.slice/crio-3b230feffece7452e111d0fa5d2ed744d8247c2cce6578ba76fb6d384bd2c565 WatchSource:0}: Error finding container 3b230feffece7452e111d0fa5d2ed744d8247c2cce6578ba76fb6d384bd2c565: Status 404 returned error can't find the container with id 3b230feffece7452e111d0fa5d2ed744d8247c2cce6578ba76fb6d384bd2c565 Sep 17 15:37:34.216615 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:34.216592 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nw49b" Sep 17 15:37:34.503530 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:34.503506 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nw49b"] Sep 17 15:37:34.505112 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:37:34.505086 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaeca8ea6_c974_404b_9c84_ea9a2c5e8f5e.slice/crio-b52a15ba178401e07cd89a764e8353733145ccd4feb52046bc4d1637f2995836 WatchSource:0}: Error finding container b52a15ba178401e07cd89a764e8353733145ccd4feb52046bc4d1637f2995836: Status 404 returned error can't find the container with id b52a15ba178401e07cd89a764e8353733145ccd4feb52046bc4d1637f2995836 Sep 17 15:37:34.679249 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:34.679213 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nw49b" event={"ID":"aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e","Type":"ContainerStarted","Data":"b52a15ba178401e07cd89a764e8353733145ccd4feb52046bc4d1637f2995836"} Sep 17 15:37:34.680309 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:34.680278 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-r7kjt" event={"ID":"c91d0865-e184-4989-8bde-d349073aefc3","Type":"ContainerStarted","Data":"3b230feffece7452e111d0fa5d2ed744d8247c2cce6578ba76fb6d384bd2c565"} Sep 17 15:37:37.697859 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:37.697827 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nw49b" event={"ID":"aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e","Type":"ContainerStarted","Data":"2ec2f5eb2eddacfdf6471184b277bef02da88023db21857f8c2c988c4c4f00d5"} Sep 17 15:37:37.699365 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:37.699331 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-r7kjt" event={"ID":"c91d0865-e184-4989-8bde-d349073aefc3","Type":"ContainerStarted","Data":"c69f806eb72f64b96d2b2dc0bd3ac8950348fe9f9f17ff41b3fe05ab456d61d4"} Sep 17 15:37:37.722464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:37.722413 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-nw49b" podStartSLOduration=2.422438816 podStartE2EDuration="4.722398431s" totalImagesPullingTime="2.299959615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:37:37.720864279 +0000 UTC m=+444.717101037" watchObservedRunningTime="2026-09-17 15:37:37.722398431 +0000 UTC m=+444.718635177" Sep 17 15:37:37.749906 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:37.749845 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-r7kjt" podStartSLOduration=2.233599581 podStartE2EDuration="4.749826962s" totalImagesPullingTime="2.516227381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:37:37.749815467 +0000 UTC m=+444.746052214" watchObservedRunningTime="2026-09-17 15:37:37.749826962 +0000 UTC m=+444.746063709" Sep 17 15:37:37.773382 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:37.773349 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-r7kjt"] Sep 17 15:37:39.708261 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:39.708215 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-r7kjt" podUID="c91d0865-e184-4989-8bde-d349073aefc3" containerName="authorino" containerID="cri-o://c69f806eb72f64b96d2b2dc0bd3ac8950348fe9f9f17ff41b3fe05ab456d61d4" gracePeriod=30 Sep 17 15:37:39.944046 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:39.944022 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-r7kjt" Sep 17 15:37:39.983429 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:39.983374 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48mlr\" (UniqueName: \"kubernetes.io/projected/c91d0865-e184-4989-8bde-d349073aefc3-kube-api-access-48mlr\") pod \"c91d0865-e184-4989-8bde-d349073aefc3\" (UID: \"c91d0865-e184-4989-8bde-d349073aefc3\") " Sep 17 15:37:39.985613 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:39.985584 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91d0865-e184-4989-8bde-d349073aefc3-kube-api-access-48mlr" (OuterVolumeSpecName: "kube-api-access-48mlr") pod "c91d0865-e184-4989-8bde-d349073aefc3" (UID: "c91d0865-e184-4989-8bde-d349073aefc3"). InnerVolumeSpecName "kube-api-access-48mlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:37:40.084454 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:40.084428 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48mlr\" (UniqueName: \"kubernetes.io/projected/c91d0865-e184-4989-8bde-d349073aefc3-kube-api-access-48mlr\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:37:40.714728 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:40.714693 2581 generic.go:356] "Generic (PLEG): container finished" podID="c91d0865-e184-4989-8bde-d349073aefc3" containerID="c69f806eb72f64b96d2b2dc0bd3ac8950348fe9f9f17ff41b3fe05ab456d61d4" exitCode=0 Sep 17 15:37:40.715209 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:40.714737 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-r7kjt" Sep 17 15:37:40.715209 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:40.714781 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-r7kjt" event={"ID":"c91d0865-e184-4989-8bde-d349073aefc3","Type":"ContainerDied","Data":"c69f806eb72f64b96d2b2dc0bd3ac8950348fe9f9f17ff41b3fe05ab456d61d4"} Sep 17 15:37:40.715209 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:40.714828 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-r7kjt" event={"ID":"c91d0865-e184-4989-8bde-d349073aefc3","Type":"ContainerDied","Data":"3b230feffece7452e111d0fa5d2ed744d8247c2cce6578ba76fb6d384bd2c565"} Sep 17 15:37:40.715209 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:40.714852 2581 scope.go:117] "RemoveContainer" containerID="c69f806eb72f64b96d2b2dc0bd3ac8950348fe9f9f17ff41b3fe05ab456d61d4" Sep 17 15:37:40.723714 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:40.723699 2581 scope.go:117] "RemoveContainer" containerID="c69f806eb72f64b96d2b2dc0bd3ac8950348fe9f9f17ff41b3fe05ab456d61d4" Sep 17 15:37:40.723977 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:37:40.723960 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69f806eb72f64b96d2b2dc0bd3ac8950348fe9f9f17ff41b3fe05ab456d61d4\": container with ID starting with c69f806eb72f64b96d2b2dc0bd3ac8950348fe9f9f17ff41b3fe05ab456d61d4 not found: ID does not exist" containerID="c69f806eb72f64b96d2b2dc0bd3ac8950348fe9f9f17ff41b3fe05ab456d61d4" Sep 17 15:37:40.724037 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:40.723984 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69f806eb72f64b96d2b2dc0bd3ac8950348fe9f9f17ff41b3fe05ab456d61d4"} err="failed to get container status \"c69f806eb72f64b96d2b2dc0bd3ac8950348fe9f9f17ff41b3fe05ab456d61d4\": rpc error: code = NotFound desc = could not find container \"c69f806eb72f64b96d2b2dc0bd3ac8950348fe9f9f17ff41b3fe05ab456d61d4\": container with ID starting with c69f806eb72f64b96d2b2dc0bd3ac8950348fe9f9f17ff41b3fe05ab456d61d4 not found: ID does not exist" Sep 17 15:37:40.741206 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:40.741184 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-r7kjt"] Sep 17 15:37:40.745022 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:40.745005 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-r7kjt"] Sep 17 15:37:41.461383 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:41.461353 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c91d0865-e184-4989-8bde-d349073aefc3" path="/var/lib/kubelet/pods/c91d0865-e184-4989-8bde-d349073aefc3/volumes" Sep 17 15:37:48.915844 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:48.915810 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:49.146034 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:49.145987 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:37:49.401521 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:37:49.401490 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:01.948766 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:01.948732 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:02.146031 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.145991 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:02.208772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.208696 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-h2f9s"] Sep 17 15:38:02.209069 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.209042 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c91d0865-e184-4989-8bde-d349073aefc3" containerName="authorino" Sep 17 15:38:02.209069 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.209060 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c91d0865-e184-4989-8bde-d349073aefc3" containerName="authorino" Sep 17 15:38:02.209234 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.209124 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c91d0865-e184-4989-8bde-d349073aefc3" containerName="authorino" Sep 17 15:38:02.213675 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.213656 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-h2f9s" Sep 17 15:38:02.228964 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.228940 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-h2f9s"] Sep 17 15:38:02.250194 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.250163 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65h7c\" (UniqueName: \"kubernetes.io/projected/182bc5a6-7ea9-45a1-bf67-3a38ea9708af-kube-api-access-65h7c\") pod \"authorino-557f876f69-h2f9s\" (UID: \"182bc5a6-7ea9-45a1-bf67-3a38ea9708af\") " pod="kuadrant-system/authorino-557f876f69-h2f9s" Sep 17 15:38:02.279383 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.279347 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-h2f9s"] Sep 17 15:38:02.279618 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:38:02.279596 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-65h7c], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-h2f9s" podUID="182bc5a6-7ea9-45a1-bf67-3a38ea9708af" Sep 17 15:38:02.321059 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.321030 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-l9qsg"] Sep 17 15:38:02.324025 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.324010 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-l9qsg" Sep 17 15:38:02.327291 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.327269 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 17 15:38:02.337982 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.337942 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-l9qsg"] Sep 17 15:38:02.350804 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.350779 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65h7c\" (UniqueName: \"kubernetes.io/projected/182bc5a6-7ea9-45a1-bf67-3a38ea9708af-kube-api-access-65h7c\") pod \"authorino-557f876f69-h2f9s\" (UID: \"182bc5a6-7ea9-45a1-bf67-3a38ea9708af\") " pod="kuadrant-system/authorino-557f876f69-h2f9s" Sep 17 15:38:02.366667 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.366636 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65h7c\" (UniqueName: \"kubernetes.io/projected/182bc5a6-7ea9-45a1-bf67-3a38ea9708af-kube-api-access-65h7c\") pod \"authorino-557f876f69-h2f9s\" (UID: \"182bc5a6-7ea9-45a1-bf67-3a38ea9708af\") " pod="kuadrant-system/authorino-557f876f69-h2f9s" Sep 17 15:38:02.452209 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.452180 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f522667d-9fa6-46bc-b5c3-077171a7d0e4-tls-cert\") pod \"authorino-6d4dfc649f-l9qsg\" (UID: \"f522667d-9fa6-46bc-b5c3-077171a7d0e4\") " pod="kuadrant-system/authorino-6d4dfc649f-l9qsg" Sep 17 15:38:02.452370 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.452253 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77q5\" (UniqueName: \"kubernetes.io/projected/f522667d-9fa6-46bc-b5c3-077171a7d0e4-kube-api-access-t77q5\") pod \"authorino-6d4dfc649f-l9qsg\" (UID: \"f522667d-9fa6-46bc-b5c3-077171a7d0e4\") " pod="kuadrant-system/authorino-6d4dfc649f-l9qsg" Sep 17 15:38:02.476554 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.476494 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-l9qsg"] Sep 17 15:38:02.476723 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:38:02.476706 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-t77q5 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-l9qsg" podUID="f522667d-9fa6-46bc-b5c3-077171a7d0e4" Sep 17 15:38:02.507083 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.507052 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65b5b59776-2mvvn"] Sep 17 15:38:02.510306 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.510289 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65b5b59776-2mvvn" Sep 17 15:38:02.526879 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.526852 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65b5b59776-2mvvn"] Sep 17 15:38:02.553339 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.553310 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t77q5\" (UniqueName: \"kubernetes.io/projected/f522667d-9fa6-46bc-b5c3-077171a7d0e4-kube-api-access-t77q5\") pod \"authorino-6d4dfc649f-l9qsg\" (UID: \"f522667d-9fa6-46bc-b5c3-077171a7d0e4\") " pod="kuadrant-system/authorino-6d4dfc649f-l9qsg" Sep 17 15:38:02.553467 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.553360 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsv2b\" (UniqueName: \"kubernetes.io/projected/dcd425bb-0b7c-4b29-8d78-f73b198dc835-kube-api-access-rsv2b\") pod \"authorino-65b5b59776-2mvvn\" (UID: \"dcd425bb-0b7c-4b29-8d78-f73b198dc835\") " pod="kuadrant-system/authorino-65b5b59776-2mvvn" Sep 17 15:38:02.553467 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.553388 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f522667d-9fa6-46bc-b5c3-077171a7d0e4-tls-cert\") pod \"authorino-6d4dfc649f-l9qsg\" (UID: \"f522667d-9fa6-46bc-b5c3-077171a7d0e4\") " pod="kuadrant-system/authorino-6d4dfc649f-l9qsg" Sep 17 15:38:02.553467 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.553418 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dcd425bb-0b7c-4b29-8d78-f73b198dc835-tls-cert\") pod \"authorino-65b5b59776-2mvvn\" (UID: \"dcd425bb-0b7c-4b29-8d78-f73b198dc835\") " pod="kuadrant-system/authorino-65b5b59776-2mvvn" Sep 17 15:38:02.555737 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.555712 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f522667d-9fa6-46bc-b5c3-077171a7d0e4-tls-cert\") pod \"authorino-6d4dfc649f-l9qsg\" (UID: \"f522667d-9fa6-46bc-b5c3-077171a7d0e4\") " pod="kuadrant-system/authorino-6d4dfc649f-l9qsg" Sep 17 15:38:02.567201 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.567180 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77q5\" (UniqueName: \"kubernetes.io/projected/f522667d-9fa6-46bc-b5c3-077171a7d0e4-kube-api-access-t77q5\") pod \"authorino-6d4dfc649f-l9qsg\" (UID: \"f522667d-9fa6-46bc-b5c3-077171a7d0e4\") " pod="kuadrant-system/authorino-6d4dfc649f-l9qsg" Sep 17 15:38:02.654461 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.654425 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsv2b\" (UniqueName: \"kubernetes.io/projected/dcd425bb-0b7c-4b29-8d78-f73b198dc835-kube-api-access-rsv2b\") pod \"authorino-65b5b59776-2mvvn\" (UID: \"dcd425bb-0b7c-4b29-8d78-f73b198dc835\") " pod="kuadrant-system/authorino-65b5b59776-2mvvn" Sep 17 15:38:02.654575 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.654470 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dcd425bb-0b7c-4b29-8d78-f73b198dc835-tls-cert\") pod \"authorino-65b5b59776-2mvvn\" (UID: \"dcd425bb-0b7c-4b29-8d78-f73b198dc835\") " pod="kuadrant-system/authorino-65b5b59776-2mvvn" Sep 17 15:38:02.656624 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.656603 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dcd425bb-0b7c-4b29-8d78-f73b198dc835-tls-cert\") pod \"authorino-65b5b59776-2mvvn\" (UID: \"dcd425bb-0b7c-4b29-8d78-f73b198dc835\") " pod="kuadrant-system/authorino-65b5b59776-2mvvn" Sep 17 15:38:02.663926 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.663897 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsv2b\" (UniqueName: \"kubernetes.io/projected/dcd425bb-0b7c-4b29-8d78-f73b198dc835-kube-api-access-rsv2b\") pod \"authorino-65b5b59776-2mvvn\" (UID: \"dcd425bb-0b7c-4b29-8d78-f73b198dc835\") " pod="kuadrant-system/authorino-65b5b59776-2mvvn" Sep 17 15:38:02.820706 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.820673 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65b5b59776-2mvvn" Sep 17 15:38:02.829688 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.829663 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-l9qsg" Sep 17 15:38:02.829792 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.829663 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-h2f9s" Sep 17 15:38:02.838799 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.838774 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-l9qsg" Sep 17 15:38:02.841688 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.841670 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-h2f9s" Sep 17 15:38:02.956285 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.956252 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65h7c\" (UniqueName: \"kubernetes.io/projected/182bc5a6-7ea9-45a1-bf67-3a38ea9708af-kube-api-access-65h7c\") pod \"182bc5a6-7ea9-45a1-bf67-3a38ea9708af\" (UID: \"182bc5a6-7ea9-45a1-bf67-3a38ea9708af\") " Sep 17 15:38:02.956664 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.956298 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f522667d-9fa6-46bc-b5c3-077171a7d0e4-tls-cert\") pod \"f522667d-9fa6-46bc-b5c3-077171a7d0e4\" (UID: \"f522667d-9fa6-46bc-b5c3-077171a7d0e4\") " Sep 17 15:38:02.956664 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.956346 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t77q5\" (UniqueName: \"kubernetes.io/projected/f522667d-9fa6-46bc-b5c3-077171a7d0e4-kube-api-access-t77q5\") pod \"f522667d-9fa6-46bc-b5c3-077171a7d0e4\" (UID: \"f522667d-9fa6-46bc-b5c3-077171a7d0e4\") " Sep 17 15:38:02.958260 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.958232 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182bc5a6-7ea9-45a1-bf67-3a38ea9708af-kube-api-access-65h7c" (OuterVolumeSpecName: "kube-api-access-65h7c") pod "182bc5a6-7ea9-45a1-bf67-3a38ea9708af" (UID: "182bc5a6-7ea9-45a1-bf67-3a38ea9708af"). InnerVolumeSpecName "kube-api-access-65h7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:38:02.958336 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.958320 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f522667d-9fa6-46bc-b5c3-077171a7d0e4-kube-api-access-t77q5" (OuterVolumeSpecName: "kube-api-access-t77q5") pod "f522667d-9fa6-46bc-b5c3-077171a7d0e4" (UID: "f522667d-9fa6-46bc-b5c3-077171a7d0e4"). InnerVolumeSpecName "kube-api-access-t77q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:38:02.958388 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:02.958370 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f522667d-9fa6-46bc-b5c3-077171a7d0e4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f522667d-9fa6-46bc-b5c3-077171a7d0e4" (UID: "f522667d-9fa6-46bc-b5c3-077171a7d0e4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:38:03.057303 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.057276 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65h7c\" (UniqueName: \"kubernetes.io/projected/182bc5a6-7ea9-45a1-bf67-3a38ea9708af-kube-api-access-65h7c\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:38:03.057303 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.057301 2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f522667d-9fa6-46bc-b5c3-077171a7d0e4-tls-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:38:03.057446 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.057315 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t77q5\" (UniqueName: \"kubernetes.io/projected/f522667d-9fa6-46bc-b5c3-077171a7d0e4-kube-api-access-t77q5\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:38:03.098723 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.098699 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65b5b59776-2mvvn"] Sep 17 15:38:03.100711 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:38:03.100685 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcd425bb_0b7c_4b29_8d78_f73b198dc835.slice/crio-7a15735a6b907389ca967cc6ea3995c5dd8b6ff9391e67b8b4c73c32fda07232 WatchSource:0}: Error finding container 7a15735a6b907389ca967cc6ea3995c5dd8b6ff9391e67b8b4c73c32fda07232: Status 404 returned error can't find the container with id 7a15735a6b907389ca967cc6ea3995c5dd8b6ff9391e67b8b4c73c32fda07232 Sep 17 15:38:03.835671 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.835638 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65b5b59776-2mvvn" event={"ID":"dcd425bb-0b7c-4b29-8d78-f73b198dc835","Type":"ContainerStarted","Data":"ab6841b4be085592b0b6b78eaecc4c636b8a994c48e3deb7dc7d1f4e3a9bf7ff"} Sep 17 15:38:03.835671 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.835673 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-l9qsg" Sep 17 15:38:03.835933 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.835683 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65b5b59776-2mvvn" event={"ID":"dcd425bb-0b7c-4b29-8d78-f73b198dc835","Type":"ContainerStarted","Data":"7a15735a6b907389ca967cc6ea3995c5dd8b6ff9391e67b8b4c73c32fda07232"} Sep 17 15:38:03.835933 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.835768 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-h2f9s" Sep 17 15:38:03.860471 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.860421 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65b5b59776-2mvvn" podStartSLOduration=1.337418292 podStartE2EDuration="1.860405989s" totalImagesPullingTime="522.987697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:03.858443056 +0000 UTC m=+470.854679801" watchObservedRunningTime="2026-09-17 15:38:03.860405989 +0000 UTC m=+470.856642735" Sep 17 15:38:03.902266 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.902192 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nw49b"] Sep 17 15:38:03.902421 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.902364 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-nw49b" podUID="aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e" containerName="authorino" containerID="cri-o://2ec2f5eb2eddacfdf6471184b277bef02da88023db21857f8c2c988c4c4f00d5" gracePeriod=30 Sep 17 15:38:03.910008 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.909981 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-h2f9s"] Sep 17 15:38:03.921188 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.921163 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-h2f9s"] Sep 17 15:38:03.952407 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.952381 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-l9qsg"] Sep 17 15:38:03.968435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:03.968407 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-l9qsg"] Sep 17 15:38:04.127179 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:04.127154 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nw49b" Sep 17 15:38:04.166337 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:04.166270 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjjrq\" (UniqueName: \"kubernetes.io/projected/aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e-kube-api-access-cjjrq\") pod \"aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e\" (UID: \"aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e\") " Sep 17 15:38:04.171709 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:04.169122 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e-kube-api-access-cjjrq" (OuterVolumeSpecName: "kube-api-access-cjjrq") pod "aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e" (UID: "aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e"). InnerVolumeSpecName "kube-api-access-cjjrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:38:04.267710 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:04.267684 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjjrq\" (UniqueName: \"kubernetes.io/projected/aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e-kube-api-access-cjjrq\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:38:04.842248 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:04.842211 2581 generic.go:356] "Generic (PLEG): container finished" podID="aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e" containerID="2ec2f5eb2eddacfdf6471184b277bef02da88023db21857f8c2c988c4c4f00d5" exitCode=0 Sep 17 15:38:04.842435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:04.842264 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nw49b" Sep 17 15:38:04.842435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:04.842294 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nw49b" event={"ID":"aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e","Type":"ContainerDied","Data":"2ec2f5eb2eddacfdf6471184b277bef02da88023db21857f8c2c988c4c4f00d5"} Sep 17 15:38:04.842435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:04.842334 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nw49b" event={"ID":"aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e","Type":"ContainerDied","Data":"b52a15ba178401e07cd89a764e8353733145ccd4feb52046bc4d1637f2995836"} Sep 17 15:38:04.842435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:04.842351 2581 scope.go:117] "RemoveContainer" containerID="2ec2f5eb2eddacfdf6471184b277bef02da88023db21857f8c2c988c4c4f00d5" Sep 17 15:38:04.851235 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:04.851217 2581 scope.go:117] "RemoveContainer" containerID="2ec2f5eb2eddacfdf6471184b277bef02da88023db21857f8c2c988c4c4f00d5" Sep 17 15:38:04.851481 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:38:04.851461 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec2f5eb2eddacfdf6471184b277bef02da88023db21857f8c2c988c4c4f00d5\": container with ID starting with 2ec2f5eb2eddacfdf6471184b277bef02da88023db21857f8c2c988c4c4f00d5 not found: ID does not exist" containerID="2ec2f5eb2eddacfdf6471184b277bef02da88023db21857f8c2c988c4c4f00d5" Sep 17 15:38:04.851529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:04.851489 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec2f5eb2eddacfdf6471184b277bef02da88023db21857f8c2c988c4c4f00d5"} err="failed to get container status \"2ec2f5eb2eddacfdf6471184b277bef02da88023db21857f8c2c988c4c4f00d5\": rpc error: code = NotFound desc = could not find container \"2ec2f5eb2eddacfdf6471184b277bef02da88023db21857f8c2c988c4c4f00d5\": container with ID starting with 2ec2f5eb2eddacfdf6471184b277bef02da88023db21857f8c2c988c4c4f00d5 not found: ID does not exist" Sep 17 15:38:04.868850 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:04.868828 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nw49b"] Sep 17 15:38:04.875480 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:04.875459 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nw49b"] Sep 17 15:38:05.461813 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:05.461780 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="182bc5a6-7ea9-45a1-bf67-3a38ea9708af" path="/var/lib/kubelet/pods/182bc5a6-7ea9-45a1-bf67-3a38ea9708af/volumes" Sep 17 15:38:05.462175 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:05.462012 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e" path="/var/lib/kubelet/pods/aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e/volumes" Sep 17 15:38:05.462317 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:05.462305 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f522667d-9fa6-46bc-b5c3-077171a7d0e4" path="/var/lib/kubelet/pods/f522667d-9fa6-46bc-b5c3-077171a7d0e4/volumes" Sep 17 15:38:14.041525 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:14.041485 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:14.126565 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:14.126524 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:14.201470 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:14.201434 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:14.286078 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:14.286048 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:14.374945 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:14.374851 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:17.862712 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:17.862675 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:18.049686 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:18.049656 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:49.146611 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:49.146579 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:49.342453 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:49.342426 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:50.818898 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:50.818867 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:50.946075 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:50.946045 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:51.428814 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:51.428762 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:51.531409 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:51.531382 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:38:51.749158 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:38:51.749050 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:39:02.914583 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:02.914547 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:39:03.011506 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:03.011472 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:39:08.076129 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:08.076023 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:39:13.916466 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:13.916422 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:39:18.278352 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.278318 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt"] Sep 17 15:39:18.278807 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.278707 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e" containerName="authorino" Sep 17 15:39:18.278807 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.278722 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e" containerName="authorino" Sep 17 15:39:18.278807 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.278773 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeca8ea6-c974-404b-9c84-ea9a2c5e8f5e" containerName="authorino" Sep 17 15:39:18.281471 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.281451 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.285423 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.285401 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 17 15:39:18.285519 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.285463 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 17 15:39:18.286222 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.286204 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 17 15:39:18.286322 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.286254 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ndjls\"" Sep 17 15:39:18.297619 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.297590 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt"] Sep 17 15:39:18.331529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.331506 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:39:18.346648 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.346625 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/226f41ff-3630-4cb3-9635-64f6cc9cc16d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.346731 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.346670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qszz\" (UniqueName: \"kubernetes.io/projected/226f41ff-3630-4cb3-9635-64f6cc9cc16d-kube-api-access-4qszz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.346731 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.346693 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/226f41ff-3630-4cb3-9635-64f6cc9cc16d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.346824 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.346736 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226f41ff-3630-4cb3-9635-64f6cc9cc16d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.346824 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.346757 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/226f41ff-3630-4cb3-9635-64f6cc9cc16d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.346824 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.346779 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/226f41ff-3630-4cb3-9635-64f6cc9cc16d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.447664 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.447637 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qszz\" (UniqueName: \"kubernetes.io/projected/226f41ff-3630-4cb3-9635-64f6cc9cc16d-kube-api-access-4qszz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.447793 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.447670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/226f41ff-3630-4cb3-9635-64f6cc9cc16d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.447793 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.447690 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226f41ff-3630-4cb3-9635-64f6cc9cc16d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.447793 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.447710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/226f41ff-3630-4cb3-9635-64f6cc9cc16d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.447983 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.447841 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/226f41ff-3630-4cb3-9635-64f6cc9cc16d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.447983 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.447889 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/226f41ff-3630-4cb3-9635-64f6cc9cc16d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.448146 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.448120 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/226f41ff-3630-4cb3-9635-64f6cc9cc16d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.448146 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.448135 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226f41ff-3630-4cb3-9635-64f6cc9cc16d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.448296 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.448253 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/226f41ff-3630-4cb3-9635-64f6cc9cc16d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.449931 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.449896 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/226f41ff-3630-4cb3-9635-64f6cc9cc16d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.450254 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.450237 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/226f41ff-3630-4cb3-9635-64f6cc9cc16d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.461184 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.461157 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qszz\" (UniqueName: \"kubernetes.io/projected/226f41ff-3630-4cb3-9635-64f6cc9cc16d-kube-api-access-4qszz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x8rtt\" (UID: \"226f41ff-3630-4cb3-9635-64f6cc9cc16d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.592846 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.592763 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:18.865039 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:18.865012 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt"] Sep 17 15:39:18.866807 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:39:18.866779 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226f41ff_3630_4cb3_9635_64f6cc9cc16d.slice/crio-f106b6d208199b889e37d8a165283451e7c908bd44f11d33652621af39148e4c WatchSource:0}: Error finding container f106b6d208199b889e37d8a165283451e7c908bd44f11d33652621af39148e4c: Status 404 returned error can't find the container with id f106b6d208199b889e37d8a165283451e7c908bd44f11d33652621af39148e4c Sep 17 15:39:19.193765 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:19.193686 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" event={"ID":"226f41ff-3630-4cb3-9635-64f6cc9cc16d","Type":"ContainerStarted","Data":"f106b6d208199b889e37d8a165283451e7c908bd44f11d33652621af39148e4c"} Sep 17 15:39:20.199459 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:20.199363 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" event={"ID":"226f41ff-3630-4cb3-9635-64f6cc9cc16d","Type":"ContainerStarted","Data":"415a6d6d5a12efc0936a2f5a2d31eb1a9f0f5348d5b7ee2e93adc847212df61c"} Sep 17 15:39:20.219635 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:20.219580 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" podStartSLOduration=1.1959190020000001 podStartE2EDuration="2.219562271s" totalImagesPullingTime="1.023643269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:20.217894985 +0000 UTC m=+547.214131727" watchObservedRunningTime="2026-09-17 15:39:20.219562271 +0000 UTC m=+547.215799016" Sep 17 15:39:27.205957 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:27.205904 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:39:28.593248 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:28.593205 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:28.593696 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:28.593286 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:28.605858 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:28.605834 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:29.257161 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:29.257135 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x8rtt" Sep 17 15:39:39.299672 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:39.299639 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:39:47.182125 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:47.182087 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:39:47.297162 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:47.297125 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:39:50.593004 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:50.592972 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:39:51.745255 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:51.745218 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:39:51.829098 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:39:51.829074 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:40:01.582376 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:40:01.582341 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:40:10.891821 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:40:10.891787 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:40:22.260793 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:40:22.260758 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:40:24.693175 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:40:24.693140 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:41:13.479514 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:13.479476 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:41:33.872066 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:33.872025 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:41:57.179516 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:57.179475 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f85fc86df-hl4bd"] Sep 17 15:41:57.182570 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:57.182553 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f85fc86df-hl4bd" Sep 17 15:41:57.191233 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:57.191205 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f85fc86df-hl4bd"] Sep 17 15:41:57.242468 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:57.242435 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3898c87d-f4a6-4ce0-8190-a08ed2ca28e9-tls-cert\") pod \"authorino-6f85fc86df-hl4bd\" (UID: \"3898c87d-f4a6-4ce0-8190-a08ed2ca28e9\") " pod="kuadrant-system/authorino-6f85fc86df-hl4bd" Sep 17 15:41:57.242588 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:57.242489 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9kpn\" (UniqueName: \"kubernetes.io/projected/3898c87d-f4a6-4ce0-8190-a08ed2ca28e9-kube-api-access-n9kpn\") pod \"authorino-6f85fc86df-hl4bd\" (UID: \"3898c87d-f4a6-4ce0-8190-a08ed2ca28e9\") " pod="kuadrant-system/authorino-6f85fc86df-hl4bd" Sep 17 15:41:57.343010 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:57.342979 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9kpn\" (UniqueName: \"kubernetes.io/projected/3898c87d-f4a6-4ce0-8190-a08ed2ca28e9-kube-api-access-n9kpn\") pod \"authorino-6f85fc86df-hl4bd\" (UID: \"3898c87d-f4a6-4ce0-8190-a08ed2ca28e9\") " pod="kuadrant-system/authorino-6f85fc86df-hl4bd" Sep 17 15:41:57.343135 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:57.343039 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3898c87d-f4a6-4ce0-8190-a08ed2ca28e9-tls-cert\") pod \"authorino-6f85fc86df-hl4bd\" (UID: \"3898c87d-f4a6-4ce0-8190-a08ed2ca28e9\") " pod="kuadrant-system/authorino-6f85fc86df-hl4bd" Sep 17 15:41:57.345410 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:57.345391 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3898c87d-f4a6-4ce0-8190-a08ed2ca28e9-tls-cert\") pod \"authorino-6f85fc86df-hl4bd\" (UID: \"3898c87d-f4a6-4ce0-8190-a08ed2ca28e9\") " pod="kuadrant-system/authorino-6f85fc86df-hl4bd" Sep 17 15:41:57.351715 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:57.351694 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9kpn\" (UniqueName: \"kubernetes.io/projected/3898c87d-f4a6-4ce0-8190-a08ed2ca28e9-kube-api-access-n9kpn\") pod \"authorino-6f85fc86df-hl4bd\" (UID: \"3898c87d-f4a6-4ce0-8190-a08ed2ca28e9\") " pod="kuadrant-system/authorino-6f85fc86df-hl4bd" Sep 17 15:41:57.492630 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:57.492557 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f85fc86df-hl4bd" Sep 17 15:41:57.763649 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:57.763611 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f85fc86df-hl4bd"] Sep 17 15:41:57.765888 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:41:57.765853 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3898c87d_f4a6_4ce0_8190_a08ed2ca28e9.slice/crio-0cf05c6e18d3a78325f5fbccd72beb1b5d86e86d64bf3adb07b2e5f8eb3cff58 WatchSource:0}: Error finding container 0cf05c6e18d3a78325f5fbccd72beb1b5d86e86d64bf3adb07b2e5f8eb3cff58: Status 404 returned error can't find the container with id 0cf05c6e18d3a78325f5fbccd72beb1b5d86e86d64bf3adb07b2e5f8eb3cff58 Sep 17 15:41:57.767320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:57.767298 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 15:41:57.958707 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:57.958673 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f85fc86df-hl4bd" event={"ID":"3898c87d-f4a6-4ce0-8190-a08ed2ca28e9","Type":"ContainerStarted","Data":"0cf05c6e18d3a78325f5fbccd72beb1b5d86e86d64bf3adb07b2e5f8eb3cff58"} Sep 17 15:41:58.966668 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:58.966568 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f85fc86df-hl4bd" event={"ID":"3898c87d-f4a6-4ce0-8190-a08ed2ca28e9","Type":"ContainerStarted","Data":"3e25937e8e52bc9befbfbe81f9cf3b9ffd40da519c01ef9bdacb7866e906ef64"} Sep 17 15:41:58.985807 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:58.985747 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f85fc86df-hl4bd" podStartSLOduration=1.152098369 podStartE2EDuration="1.985729295s" totalImagesPullingTime="833.630926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:58.983515621 +0000 UTC m=+705.979752366" watchObservedRunningTime="2026-09-17 15:41:58.985729295 +0000 UTC m=+705.981966041" Sep 17 15:41:59.036854 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.036774 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65b5b59776-2mvvn"] Sep 17 15:41:59.037072 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.037040 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-65b5b59776-2mvvn" podUID="dcd425bb-0b7c-4b29-8d78-f73b198dc835" containerName="authorino" containerID="cri-o://ab6841b4be085592b0b6b78eaecc4c636b8a994c48e3deb7dc7d1f4e3a9bf7ff" gracePeriod=30 Sep 17 15:41:59.289121 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.289092 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65b5b59776-2mvvn" Sep 17 15:41:59.360036 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.359980 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsv2b\" (UniqueName: \"kubernetes.io/projected/dcd425bb-0b7c-4b29-8d78-f73b198dc835-kube-api-access-rsv2b\") pod \"dcd425bb-0b7c-4b29-8d78-f73b198dc835\" (UID: \"dcd425bb-0b7c-4b29-8d78-f73b198dc835\") " Sep 17 15:41:59.360257 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.360077 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dcd425bb-0b7c-4b29-8d78-f73b198dc835-tls-cert\") pod \"dcd425bb-0b7c-4b29-8d78-f73b198dc835\" (UID: \"dcd425bb-0b7c-4b29-8d78-f73b198dc835\") " Sep 17 15:41:59.362262 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.362217 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd425bb-0b7c-4b29-8d78-f73b198dc835-kube-api-access-rsv2b" (OuterVolumeSpecName: "kube-api-access-rsv2b") pod "dcd425bb-0b7c-4b29-8d78-f73b198dc835" (UID: "dcd425bb-0b7c-4b29-8d78-f73b198dc835"). InnerVolumeSpecName "kube-api-access-rsv2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:41:59.369820 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.369795 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd425bb-0b7c-4b29-8d78-f73b198dc835-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "dcd425bb-0b7c-4b29-8d78-f73b198dc835" (UID: "dcd425bb-0b7c-4b29-8d78-f73b198dc835"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:41:59.461267 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.461226 2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dcd425bb-0b7c-4b29-8d78-f73b198dc835-tls-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:41:59.461267 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.461253 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsv2b\" (UniqueName: \"kubernetes.io/projected/dcd425bb-0b7c-4b29-8d78-f73b198dc835-kube-api-access-rsv2b\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:41:59.599238 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.599165 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:41:59.702905 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.702833 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:41:59.975168 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.975079 2581 generic.go:356] "Generic (PLEG): container finished" podID="dcd425bb-0b7c-4b29-8d78-f73b198dc835" containerID="ab6841b4be085592b0b6b78eaecc4c636b8a994c48e3deb7dc7d1f4e3a9bf7ff" exitCode=0 Sep 17 15:41:59.975168 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.975141 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65b5b59776-2mvvn" Sep 17 15:41:59.975168 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.975161 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65b5b59776-2mvvn" event={"ID":"dcd425bb-0b7c-4b29-8d78-f73b198dc835","Type":"ContainerDied","Data":"ab6841b4be085592b0b6b78eaecc4c636b8a994c48e3deb7dc7d1f4e3a9bf7ff"} Sep 17 15:41:59.975732 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.975196 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65b5b59776-2mvvn" event={"ID":"dcd425bb-0b7c-4b29-8d78-f73b198dc835","Type":"ContainerDied","Data":"7a15735a6b907389ca967cc6ea3995c5dd8b6ff9391e67b8b4c73c32fda07232"} Sep 17 15:41:59.975732 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.975216 2581 scope.go:117] "RemoveContainer" containerID="ab6841b4be085592b0b6b78eaecc4c636b8a994c48e3deb7dc7d1f4e3a9bf7ff" Sep 17 15:41:59.985816 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.985793 2581 scope.go:117] "RemoveContainer" containerID="ab6841b4be085592b0b6b78eaecc4c636b8a994c48e3deb7dc7d1f4e3a9bf7ff" Sep 17 15:41:59.987191 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:41:59.987161 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6841b4be085592b0b6b78eaecc4c636b8a994c48e3deb7dc7d1f4e3a9bf7ff\": container with ID starting with ab6841b4be085592b0b6b78eaecc4c636b8a994c48e3deb7dc7d1f4e3a9bf7ff not found: ID does not exist" containerID="ab6841b4be085592b0b6b78eaecc4c636b8a994c48e3deb7dc7d1f4e3a9bf7ff" Sep 17 15:41:59.987628 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:41:59.987596 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6841b4be085592b0b6b78eaecc4c636b8a994c48e3deb7dc7d1f4e3a9bf7ff"} err="failed to get container status \"ab6841b4be085592b0b6b78eaecc4c636b8a994c48e3deb7dc7d1f4e3a9bf7ff\": rpc error: code = NotFound desc = could not find container \"ab6841b4be085592b0b6b78eaecc4c636b8a994c48e3deb7dc7d1f4e3a9bf7ff\": container with ID starting with ab6841b4be085592b0b6b78eaecc4c636b8a994c48e3deb7dc7d1f4e3a9bf7ff not found: ID does not exist" Sep 17 15:42:00.005002 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:00.004968 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65b5b59776-2mvvn"] Sep 17 15:42:00.007986 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:00.007960 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65b5b59776-2mvvn"] Sep 17 15:42:00.834533 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:00.834506 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:00.938467 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:00.938432 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:01.461466 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:01.461432 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcd425bb-0b7c-4b29-8d78-f73b198dc835" path="/var/lib/kubelet/pods/dcd425bb-0b7c-4b29-8d78-f73b198dc835/volumes" Sep 17 15:42:21.238296 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.238263 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:21.435607 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.435563 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:21.599717 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.599686 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c"] Sep 17 15:42:21.600267 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.600147 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd425bb-0b7c-4b29-8d78-f73b198dc835" containerName="authorino" Sep 17 15:42:21.600267 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.600171 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd425bb-0b7c-4b29-8d78-f73b198dc835" containerName="authorino" Sep 17 15:42:21.600267 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.600256 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd425bb-0b7c-4b29-8d78-f73b198dc835" containerName="authorino" Sep 17 15:42:21.600591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.600559 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:21.604213 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.604196 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.606639 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.606616 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-849b71-openshift-default-dockercfg-jmhbk\"" Sep 17 15:42:21.606738 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.606619 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 15:42:21.622163 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.622140 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c"] Sep 17 15:42:21.706236 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.706204 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:21.745446 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.745402 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-workload-socket\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.745708 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.745460 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-credential-socket\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.745708 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.745503 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948331da-4b2c-4f6e-bf94-24551fdf86b5-istiod-ca-cert\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.745708 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.745597 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-envoy\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.745708 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.745638 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-workload-certs\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.746006 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.745738 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-data\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.746006 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.745779 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-podinfo\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.746006 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.745812 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-token\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.746006 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.745848 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8nmn\" (UniqueName: \"kubernetes.io/projected/948331da-4b2c-4f6e-bf94-24551fdf86b5-kube-api-access-h8nmn\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.838320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.838286 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:21.847061 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.847029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-data\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.847233 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.847068 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-podinfo\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.847233 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.847090 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-token\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.847353 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.847238 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8nmn\" (UniqueName: \"kubernetes.io/projected/948331da-4b2c-4f6e-bf94-24551fdf86b5-kube-api-access-h8nmn\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.847353 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.847283 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-workload-socket\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.847353 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.847325 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-credential-socket\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.847514 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.847417 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948331da-4b2c-4f6e-bf94-24551fdf86b5-istiod-ca-cert\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.847514 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.847478 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-envoy\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.847615 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.847520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-workload-certs\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.847747 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.847719 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-credential-socket\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.847892 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.847847 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-data\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.848040 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.848015 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-workload-certs\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.848169 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.848127 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-workload-socket\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.848371 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.848350 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948331da-4b2c-4f6e-bf94-24551fdf86b5-istiod-ca-cert\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.850342 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.850289 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-envoy\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.850342 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.850332 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-podinfo\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.872149 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.872123 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-token\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.872743 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.872727 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8nmn\" (UniqueName: \"kubernetes.io/projected/948331da-4b2c-4f6e-bf94-24551fdf86b5-kube-api-access-h8nmn\") pod \"e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.914437 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.914411 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:21.925954 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:21.925901 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:22.068487 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:22.068452 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:22.201132 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:22.201106 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c"] Sep 17 15:42:22.203356 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:42:22.203321 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948331da_4b2c_4f6e_bf94_24551fdf86b5.slice/crio-ba8fea65d52abb4558f85cc00f9842c9e534abc6e4cafacd70b2556a49b433f4 WatchSource:0}: Error finding container ba8fea65d52abb4558f85cc00f9842c9e534abc6e4cafacd70b2556a49b433f4: Status 404 returned error can't find the container with id ba8fea65d52abb4558f85cc00f9842c9e534abc6e4cafacd70b2556a49b433f4 Sep 17 15:42:22.754110 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:22.754055 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:22.844704 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:22.844675 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:23.087061 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:23.087026 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" event={"ID":"948331da-4b2c-4f6e-bf94-24551fdf86b5","Type":"ContainerStarted","Data":"ba8fea65d52abb4558f85cc00f9842c9e534abc6e4cafacd70b2556a49b433f4"} Sep 17 15:42:23.216518 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:23.216485 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:23.356396 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:23.356319 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7800-5a338b"] Sep 17 15:42:23.360435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:23.360411 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-5a338b" Sep 17 15:42:23.363168 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:23.363151 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-55clv\"" Sep 17 15:42:23.368705 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:23.368684 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7800-5a338b"] Sep 17 15:42:23.461263 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:23.461225 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bplxj\" (UniqueName: \"kubernetes.io/projected/d076c686-fb69-42e6-a6ca-7467aaa43ec6-kube-api-access-bplxj\") pod \"test-curl-7800-5a338b\" (UID: \"d076c686-fb69-42e6-a6ca-7467aaa43ec6\") " pod="opendatahub/test-curl-7800-5a338b" Sep 17 15:42:23.562541 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:23.562506 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bplxj\" (UniqueName: \"kubernetes.io/projected/d076c686-fb69-42e6-a6ca-7467aaa43ec6-kube-api-access-bplxj\") pod \"test-curl-7800-5a338b\" (UID: \"d076c686-fb69-42e6-a6ca-7467aaa43ec6\") " pod="opendatahub/test-curl-7800-5a338b" Sep 17 15:42:23.572053 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:23.572028 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bplxj\" (UniqueName: \"kubernetes.io/projected/d076c686-fb69-42e6-a6ca-7467aaa43ec6-kube-api-access-bplxj\") pod \"test-curl-7800-5a338b\" (UID: \"d076c686-fb69-42e6-a6ca-7467aaa43ec6\") " pod="opendatahub/test-curl-7800-5a338b" Sep 17 15:42:23.671870 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:23.671784 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-5a338b" Sep 17 15:42:23.946109 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:23.946084 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7800-5a338b"] Sep 17 15:42:23.948962 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:42:23.948932 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd076c686_fb69_42e6_a6ca_7467aaa43ec6.slice/crio-93b072ea1796116b3e88d4fab16f36a4493589c6f1e64634c23b746fd79da491 WatchSource:0}: Error finding container 93b072ea1796116b3e88d4fab16f36a4493589c6f1e64634c23b746fd79da491: Status 404 returned error can't find the container with id 93b072ea1796116b3e88d4fab16f36a4493589c6f1e64634c23b746fd79da491 Sep 17 15:42:24.094064 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:24.094022 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7800-5a338b" event={"ID":"d076c686-fb69-42e6-a6ca-7467aaa43ec6","Type":"ContainerStarted","Data":"93b072ea1796116b3e88d4fab16f36a4493589c6f1e64634c23b746fd79da491"} Sep 17 15:42:24.941170 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:24.941126 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:25.114844 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:25.114806 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:25.291090 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:25.290991 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:26.105968 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.105935 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr"] Sep 17 15:42:26.109752 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.109729 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7800-5a338b" event={"ID":"d076c686-fb69-42e6-a6ca-7467aaa43ec6","Type":"ContainerStarted","Data":"787512b23f00934a03d7fdaddab9bf4000082f25bb667641046487a62f8db8f8"} Sep 17 15:42:26.109884 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.109806 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.113863 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.113844 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 15:42:26.113987 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.113846 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-c86e9a-serving-cert\"" Sep 17 15:42:26.114656 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.114640 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 15:42:26.114716 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.114656 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 15:42:26.114716 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.114638 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mdtj9\"" Sep 17 15:42:26.129029 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.129005 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr"] Sep 17 15:42:26.140221 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.140174 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7800-5a338b" podStartSLOduration=1.097690483 podStartE2EDuration="3.140160075s" totalImagesPullingTime="2.042469592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:26.138512695 +0000 UTC m=+733.134749438" watchObservedRunningTime="2026-09-17 15:42:26.140160075 +0000 UTC m=+733.136396821" Sep 17 15:42:26.183339 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.183312 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.183457 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.183365 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-metrics-certs\") pod \"maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.183514 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.183494 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl6qw\" (UniqueName: \"kubernetes.io/projected/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-kube-api-access-pl6qw\") pod \"maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.183553 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.183523 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.184348 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.184329 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:26.267298 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.267240 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:26.284425 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.284394 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl6qw\" (UniqueName: \"kubernetes.io/projected/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-kube-api-access-pl6qw\") pod \"maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.284515 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.284430 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.284515 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.284477 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.284515 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.284500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-metrics-certs\") pod \"maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.284665 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:42:26.284595 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-serving-cert: secret "maas-api-e2e-worker-w4-c86e9a-serving-cert" not found Sep 17 15:42:26.284719 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:42:26.284669 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-maas-api-tls podName:cf6df8da-ce1f-45b9-b416-a82cda25f5f2 nodeName:}" failed. No retries permitted until 2026-09-17 15:42:26.784651607 +0000 UTC m=+733.780888339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-maas-api-tls") pod "maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" (UID: "cf6df8da-ce1f-45b9-b416-a82cda25f5f2") : secret "maas-api-e2e-worker-w4-c86e9a-serving-cert" not found Sep 17 15:42:26.285164 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.285144 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.287049 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.287025 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-metrics-certs\") pod \"maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.299706 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.299685 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl6qw\" (UniqueName: \"kubernetes.io/projected/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-kube-api-access-pl6qw\") pod \"maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.413692 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.413663 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:26.789521 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.789485 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.791753 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.791734 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:26.856660 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.856580 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7800-5a338b"] Sep 17 15:42:26.865384 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.865354 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7800-5a338b"] Sep 17 15:42:26.909595 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:26.909563 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:27.008605 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.008572 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7800-9b635a"] Sep 17 15:42:27.008990 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.008972 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d076c686-fb69-42e6-a6ca-7467aaa43ec6" containerName="test-curl-7800-5a338b" Sep 17 15:42:27.009083 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.008992 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d076c686-fb69-42e6-a6ca-7467aaa43ec6" containerName="test-curl-7800-5a338b" Sep 17 15:42:27.009083 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.009075 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d076c686-fb69-42e6-a6ca-7467aaa43ec6" containerName="test-curl-7800-5a338b" Sep 17 15:42:27.012171 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.012150 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-9b635a" Sep 17 15:42:27.020844 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.020827 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:27.040440 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.040358 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7800-9b635a"] Sep 17 15:42:27.092663 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.092632 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlkj9\" (UniqueName: \"kubernetes.io/projected/5d76685d-0f0e-44d2-ba96-09f7d832538e-kube-api-access-xlkj9\") pod \"test-curl-7800-9b635a\" (UID: \"5d76685d-0f0e-44d2-ba96-09f7d832538e\") " pod="opendatahub/test-curl-7800-9b635a" Sep 17 15:42:27.098635 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.098612 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:27.113351 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:42:27.113297 2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7800-5a338b" container="test-curl-7800-5a338b" Sep 17 15:42:27.193099 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.193072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlkj9\" (UniqueName: \"kubernetes.io/projected/5d76685d-0f0e-44d2-ba96-09f7d832538e-kube-api-access-xlkj9\") pod \"test-curl-7800-9b635a\" (UID: \"5d76685d-0f0e-44d2-ba96-09f7d832538e\") " pod="opendatahub/test-curl-7800-9b635a" Sep 17 15:42:27.220564 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.220538 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlkj9\" (UniqueName: \"kubernetes.io/projected/5d76685d-0f0e-44d2-ba96-09f7d832538e-kube-api-access-xlkj9\") pod \"test-curl-7800-9b635a\" (UID: \"5d76685d-0f0e-44d2-ba96-09f7d832538e\") " pod="opendatahub/test-curl-7800-9b635a" Sep 17 15:42:27.321444 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.321417 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-9b635a" Sep 17 15:42:27.347621 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.347553 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr"] Sep 17 15:42:27.350232 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:42:27.350205 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6df8da_ce1f_45b9_b416_a82cda25f5f2.slice/crio-d911de2bb17d21c964a0e1d76757c64f77b59ba32c7b9225662d61caf585b91e WatchSource:0}: Error finding container d911de2bb17d21c964a0e1d76757c64f77b59ba32c7b9225662d61caf585b91e: Status 404 returned error can't find the container with id d911de2bb17d21c964a0e1d76757c64f77b59ba32c7b9225662d61caf585b91e Sep 17 15:42:27.597159 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.597137 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7800-9b635a"] Sep 17 15:42:27.598518 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:42:27.598498 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d76685d_0f0e_44d2_ba96_09f7d832538e.slice/crio-0a496bea965ac56a314c5146c23c855b7ce5a836f4216e4c5f37238b6ffd06fb WatchSource:0}: Error finding container 0a496bea965ac56a314c5146c23c855b7ce5a836f4216e4c5f37238b6ffd06fb: Status 404 returned error can't find the container with id 0a496bea965ac56a314c5146c23c855b7ce5a836f4216e4c5f37238b6ffd06fb Sep 17 15:42:27.803497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.803461 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:27.904038 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:27.903948 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:28.122515 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.122478 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7800-9b635a" event={"ID":"5d76685d-0f0e-44d2-ba96-09f7d832538e","Type":"ContainerStarted","Data":"152f7ecc34705c535b6bc61bc0cfa85d95232708e1f572316ab85bbfc29ea8d2"} Sep 17 15:42:28.122708 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.122530 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7800-9b635a" event={"ID":"5d76685d-0f0e-44d2-ba96-09f7d832538e","Type":"ContainerStarted","Data":"0a496bea965ac56a314c5146c23c855b7ce5a836f4216e4c5f37238b6ffd06fb"} Sep 17 15:42:28.125192 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.125152 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" event={"ID":"cf6df8da-ce1f-45b9-b416-a82cda25f5f2","Type":"ContainerStarted","Data":"d911de2bb17d21c964a0e1d76757c64f77b59ba32c7b9225662d61caf585b91e"} Sep 17 15:42:28.125338 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.125211 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7800-5a338b" podUID="d076c686-fb69-42e6-a6ca-7467aaa43ec6" containerName="test-curl-7800-5a338b" containerID="cri-o://787512b23f00934a03d7fdaddab9bf4000082f25bb667641046487a62f8db8f8" gracePeriod=2 Sep 17 15:42:28.159303 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.159100 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7800-9b635a" podStartSLOduration=2.159083918 podStartE2EDuration="2.159083918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:28.155615765 +0000 UTC m=+735.151852513" watchObservedRunningTime="2026-09-17 15:42:28.159083918 +0000 UTC m=+735.155320662" Sep 17 15:42:28.317732 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.317691 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:28.443508 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.443423 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:28.541850 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.541814 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:28.629638 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.629609 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr"] Sep 17 15:42:28.635454 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.635430 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" Sep 17 15:42:28.638550 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.638342 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-449c47-dockercfg-wdzbt\"" Sep 17 15:42:28.638823 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.638782 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-449c47\"" Sep 17 15:42:28.649006 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.648977 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr"] Sep 17 15:42:28.707996 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.707904 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fspnp\" (UniqueName: \"kubernetes.io/projected/8b7e10e0-a030-47f6-9a61-14d9e30093e9-kube-api-access-fspnp\") pod \"payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr\" (UID: \"8b7e10e0-a030-47f6-9a61-14d9e30093e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" Sep 17 15:42:28.707996 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.707981 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b7e10e0-a030-47f6-9a61-14d9e30093e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr\" (UID: \"8b7e10e0-a030-47f6-9a61-14d9e30093e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" Sep 17 15:42:28.719614 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.719581 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp"] Sep 17 15:42:28.726613 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.726589 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" Sep 17 15:42:28.735961 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.735876 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp"] Sep 17 15:42:28.808629 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.808597 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z458\" (UniqueName: \"kubernetes.io/projected/4ba57349-9d6c-471e-988d-7083ad61d916-kube-api-access-8z458\") pod \"payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp\" (UID: \"4ba57349-9d6c-471e-988d-7083ad61d916\") " pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" Sep 17 15:42:28.808807 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.808651 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba57349-9d6c-471e-988d-7083ad61d916-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp\" (UID: \"4ba57349-9d6c-471e-988d-7083ad61d916\") " pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" Sep 17 15:42:28.808807 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.808724 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fspnp\" (UniqueName: \"kubernetes.io/projected/8b7e10e0-a030-47f6-9a61-14d9e30093e9-kube-api-access-fspnp\") pod \"payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr\" (UID: \"8b7e10e0-a030-47f6-9a61-14d9e30093e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" Sep 17 15:42:28.808807 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.808759 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b7e10e0-a030-47f6-9a61-14d9e30093e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr\" (UID: \"8b7e10e0-a030-47f6-9a61-14d9e30093e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" Sep 17 15:42:28.809812 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.809782 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b7e10e0-a030-47f6-9a61-14d9e30093e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr\" (UID: \"8b7e10e0-a030-47f6-9a61-14d9e30093e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" Sep 17 15:42:28.818427 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.818378 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fspnp\" (UniqueName: \"kubernetes.io/projected/8b7e10e0-a030-47f6-9a61-14d9e30093e9-kube-api-access-fspnp\") pod \"payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr\" (UID: \"8b7e10e0-a030-47f6-9a61-14d9e30093e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" Sep 17 15:42:28.909357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.909315 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z458\" (UniqueName: \"kubernetes.io/projected/4ba57349-9d6c-471e-988d-7083ad61d916-kube-api-access-8z458\") pod \"payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp\" (UID: \"4ba57349-9d6c-471e-988d-7083ad61d916\") " pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" Sep 17 15:42:28.909543 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.909384 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba57349-9d6c-471e-988d-7083ad61d916-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp\" (UID: \"4ba57349-9d6c-471e-988d-7083ad61d916\") " pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" Sep 17 15:42:28.910556 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.910527 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba57349-9d6c-471e-988d-7083ad61d916-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp\" (UID: \"4ba57349-9d6c-471e-988d-7083ad61d916\") " pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" Sep 17 15:42:28.919327 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.919293 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7800-9b635a"] Sep 17 15:42:28.924615 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.924551 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7800-9b635a"] Sep 17 15:42:28.928704 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.928677 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z458\" (UniqueName: \"kubernetes.io/projected/4ba57349-9d6c-471e-988d-7083ad61d916-kube-api-access-8z458\") pod \"payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp\" (UID: \"4ba57349-9d6c-471e-988d-7083ad61d916\") " pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" Sep 17 15:42:28.948297 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.948267 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:28.958997 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:28.958898 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" Sep 17 15:42:29.047645 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:29.047605 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" Sep 17 15:42:29.133314 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:42:29.133278 2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7800-9b635a" container="test-curl-7800-9b635a" Sep 17 15:42:29.479002 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:29.478963 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:29.713282 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:29.713231 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:29.821657 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:29.821623 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:30.139951 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:30.139903 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7800-9b635a" podUID="5d76685d-0f0e-44d2-ba96-09f7d832538e" containerName="test-curl-7800-9b635a" containerID="cri-o://152f7ecc34705c535b6bc61bc0cfa85d95232708e1f572316ab85bbfc29ea8d2" gracePeriod=2 Sep 17 15:42:30.654073 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:30.654042 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:30.886341 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:30.886230 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:42:30.886341 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:30.886329 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:42:30.886476 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:30.886379 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:42:30.978494 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:30.978471 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-5a338b" Sep 17 15:42:31.130484 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.130454 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bplxj\" (UniqueName: \"kubernetes.io/projected/d076c686-fb69-42e6-a6ca-7467aaa43ec6-kube-api-access-bplxj\") pod \"d076c686-fb69-42e6-a6ca-7467aaa43ec6\" (UID: \"d076c686-fb69-42e6-a6ca-7467aaa43ec6\") " Sep 17 15:42:31.132672 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.132630 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d076c686-fb69-42e6-a6ca-7467aaa43ec6-kube-api-access-bplxj" (OuterVolumeSpecName: "kube-api-access-bplxj") pod "d076c686-fb69-42e6-a6ca-7467aaa43ec6" (UID: "d076c686-fb69-42e6-a6ca-7467aaa43ec6"). InnerVolumeSpecName "kube-api-access-bplxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:42:31.148672 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.148405 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" event={"ID":"cf6df8da-ce1f-45b9-b416-a82cda25f5f2","Type":"ContainerStarted","Data":"8176556b1872c27434e658c0b9d09bbee17a1ed581cbb970f96edbbc5cb073ee"} Sep 17 15:42:31.148672 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.148537 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:31.149856 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.149826 2581 generic.go:356] "Generic (PLEG): container finished" podID="d076c686-fb69-42e6-a6ca-7467aaa43ec6" containerID="787512b23f00934a03d7fdaddab9bf4000082f25bb667641046487a62f8db8f8" exitCode=137 Sep 17 15:42:31.149991 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.149870 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-5a338b" Sep 17 15:42:31.149991 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.149976 2581 scope.go:117] "RemoveContainer" containerID="787512b23f00934a03d7fdaddab9bf4000082f25bb667641046487a62f8db8f8" Sep 17 15:42:31.151904 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.151873 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" event={"ID":"948331da-4b2c-4f6e-bf94-24551fdf86b5","Type":"ContainerStarted","Data":"99c8b3876b11686a643bb7e1e086d91921c1364a53f09abd066149b206809734"} Sep 17 15:42:31.161080 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.161034 2581 scope.go:117] "RemoveContainer" containerID="787512b23f00934a03d7fdaddab9bf4000082f25bb667641046487a62f8db8f8" Sep 17 15:42:31.162049 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:42:31.162020 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"787512b23f00934a03d7fdaddab9bf4000082f25bb667641046487a62f8db8f8\": container with ID starting with 787512b23f00934a03d7fdaddab9bf4000082f25bb667641046487a62f8db8f8 not found: ID does not exist" containerID="787512b23f00934a03d7fdaddab9bf4000082f25bb667641046487a62f8db8f8" Sep 17 15:42:31.162156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.162058 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787512b23f00934a03d7fdaddab9bf4000082f25bb667641046487a62f8db8f8"} err="failed to get container status \"787512b23f00934a03d7fdaddab9bf4000082f25bb667641046487a62f8db8f8\": rpc error: code = NotFound desc = could not find container \"787512b23f00934a03d7fdaddab9bf4000082f25bb667641046487a62f8db8f8\": container with ID starting with 787512b23f00934a03d7fdaddab9bf4000082f25bb667641046487a62f8db8f8 not found: ID does not exist" Sep 17 15:42:31.165653 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:42:31.165633 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7e10e0_a030_47f6_9a61_14d9e30093e9.slice/crio-f6a331c6bb89e2476a6f420d501e6493b94b9fdc6c7da7becebac7b4b509eb0c WatchSource:0}: Error finding container f6a331c6bb89e2476a6f420d501e6493b94b9fdc6c7da7becebac7b4b509eb0c: Status 404 returned error can't find the container with id f6a331c6bb89e2476a6f420d501e6493b94b9fdc6c7da7becebac7b4b509eb0c Sep 17 15:42:31.165742 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.165692 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr"] Sep 17 15:42:31.182170 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.182117 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" podStartSLOduration=1.649777996 podStartE2EDuration="5.182097505s" totalImagesPullingTime="3.532319509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:31.175393261 +0000 UTC m=+738.171630007" watchObservedRunningTime="2026-09-17 15:42:31.182097505 +0000 UTC m=+738.178334251" Sep 17 15:42:31.185342 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.185324 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp"] Sep 17 15:42:31.186036 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:42:31.186016 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba57349_9d6c_471e_988d_7083ad61d916.slice/crio-d90deeadad4f187eaa752a410c6df4103cdae0466e74009c5a7855d0d17ca71d WatchSource:0}: Error finding container d90deeadad4f187eaa752a410c6df4103cdae0466e74009c5a7855d0d17ca71d: Status 404 returned error can't find the container with id d90deeadad4f187eaa752a410c6df4103cdae0466e74009c5a7855d0d17ca71d Sep 17 15:42:31.204537 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:42:31.204506 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7800-5a338b\" is forbidden: User \"system:node:ip-10-0-128-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-15.ec2.internal' and this object" podUID="d076c686-fb69-42e6-a6ca-7467aaa43ec6" pod="opendatahub/test-curl-7800-5a338b" Sep 17 15:42:31.205514 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.205478 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" podStartSLOduration=1.5256589059999999 podStartE2EDuration="10.205463453s" totalImagesPullingTime="8.679804547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:31.202528828 +0000 UTC m=+738.198765573" watchObservedRunningTime="2026-09-17 15:42:31.205463453 +0000 UTC m=+738.201700198" Sep 17 15:42:31.232080 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.232055 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bplxj\" (UniqueName: \"kubernetes.io/projected/d076c686-fb69-42e6-a6ca-7467aaa43ec6-kube-api-access-bplxj\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:42:31.399294 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.399266 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:31.462587 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.462516 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d076c686-fb69-42e6-a6ca-7467aaa43ec6" path="/var/lib/kubelet/pods/d076c686-fb69-42e6-a6ca-7467aaa43ec6/volumes" Sep 17 15:42:31.498581 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.498551 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:31.642116 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.642087 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:31.914686 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.914635 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:31.919885 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:31.919858 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:32.045066 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:32.045033 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:32.167205 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:32.167170 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" event={"ID":"4ba57349-9d6c-471e-988d-7083ad61d916","Type":"ContainerStarted","Data":"d90deeadad4f187eaa752a410c6df4103cdae0466e74009c5a7855d0d17ca71d"} Sep 17 15:42:32.170058 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:32.169984 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" event={"ID":"8b7e10e0-a030-47f6-9a61-14d9e30093e9","Type":"ContainerStarted","Data":"f6a331c6bb89e2476a6f420d501e6493b94b9fdc6c7da7becebac7b4b509eb0c"} Sep 17 15:42:32.170734 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:32.170685 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:32.172053 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:32.171820 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:42:32.384218 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:32.384180 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:32.484787 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:32.484698 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:32.620846 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:32.620815 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-9b635a" Sep 17 15:42:32.747634 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:32.747609 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlkj9\" (UniqueName: \"kubernetes.io/projected/5d76685d-0f0e-44d2-ba96-09f7d832538e-kube-api-access-xlkj9\") pod \"5d76685d-0f0e-44d2-ba96-09f7d832538e\" (UID: \"5d76685d-0f0e-44d2-ba96-09f7d832538e\") " Sep 17 15:42:32.749535 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:32.749511 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d76685d-0f0e-44d2-ba96-09f7d832538e-kube-api-access-xlkj9" (OuterVolumeSpecName: "kube-api-access-xlkj9") pod "5d76685d-0f0e-44d2-ba96-09f7d832538e" (UID: "5d76685d-0f0e-44d2-ba96-09f7d832538e"). InnerVolumeSpecName "kube-api-access-xlkj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:42:32.848478 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:32.848451 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlkj9\" (UniqueName: \"kubernetes.io/projected/5d76685d-0f0e-44d2-ba96-09f7d832538e-kube-api-access-xlkj9\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:42:33.079705 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.079667 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:33.175513 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.175475 2581 generic.go:356] "Generic (PLEG): container finished" podID="5d76685d-0f0e-44d2-ba96-09f7d832538e" containerID="152f7ecc34705c535b6bc61bc0cfa85d95232708e1f572316ab85bbfc29ea8d2" exitCode=137 Sep 17 15:42:33.175717 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.175531 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-9b635a" Sep 17 15:42:33.175717 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.175587 2581 scope.go:117] "RemoveContainer" containerID="152f7ecc34705c535b6bc61bc0cfa85d95232708e1f572316ab85bbfc29ea8d2" Sep 17 15:42:33.177441 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.177416 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:33.178354 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.177967 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" event={"ID":"4ba57349-9d6c-471e-988d-7083ad61d916","Type":"ContainerStarted","Data":"53e4b7afc913211ae4b2210f17fa635deb1d26f8169dff2bf6dd1259440c4192"} Sep 17 15:42:33.178354 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.178025 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" Sep 17 15:42:33.180386 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.180184 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" event={"ID":"8b7e10e0-a030-47f6-9a61-14d9e30093e9","Type":"ContainerStarted","Data":"024909da84d8514877fdaf74cad5d8ffa1d26e56fd57bc8b751a618f93e69c0f"} Sep 17 15:42:33.186513 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.186240 2581 scope.go:117] "RemoveContainer" containerID="152f7ecc34705c535b6bc61bc0cfa85d95232708e1f572316ab85bbfc29ea8d2" Sep 17 15:42:33.186638 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:42:33.186604 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152f7ecc34705c535b6bc61bc0cfa85d95232708e1f572316ab85bbfc29ea8d2\": container with ID starting with 152f7ecc34705c535b6bc61bc0cfa85d95232708e1f572316ab85bbfc29ea8d2 not found: ID does not exist" containerID="152f7ecc34705c535b6bc61bc0cfa85d95232708e1f572316ab85bbfc29ea8d2" Sep 17 15:42:33.186742 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.186641 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152f7ecc34705c535b6bc61bc0cfa85d95232708e1f572316ab85bbfc29ea8d2"} err="failed to get container status \"152f7ecc34705c535b6bc61bc0cfa85d95232708e1f572316ab85bbfc29ea8d2\": rpc error: code = NotFound desc = could not find container \"152f7ecc34705c535b6bc61bc0cfa85d95232708e1f572316ab85bbfc29ea8d2\": container with ID starting with 152f7ecc34705c535b6bc61bc0cfa85d95232708e1f572316ab85bbfc29ea8d2 not found: ID does not exist" Sep 17 15:42:33.203140 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.203085 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" podStartSLOduration=3.722957959 podStartE2EDuration="5.203065074s" totalImagesPullingTime="1.480107115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:33.202128582 +0000 UTC m=+740.198365331" watchObservedRunningTime="2026-09-17 15:42:33.203065074 +0000 UTC m=+740.199301820" Sep 17 15:42:33.204281 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:42:33.204255 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7800-9b635a\" is forbidden: User \"system:node:ip-10-0-128-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-15.ec2.internal' and this object" podUID="5d76685d-0f0e-44d2-ba96-09f7d832538e" pod="opendatahub/test-curl-7800-9b635a" Sep 17 15:42:33.225939 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.225881 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" podStartSLOduration=3.727982446 podStartE2EDuration="5.225862441s" totalImagesPullingTime="1.497879995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:33.22335519 +0000 UTC m=+740.219591938" watchObservedRunningTime="2026-09-17 15:42:33.225862441 +0000 UTC m=+740.222099182" Sep 17 15:42:33.463742 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.463666 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d76685d-0f0e-44d2-ba96-09f7d832538e" path="/var/lib/kubelet/pods/5d76685d-0f0e-44d2-ba96-09f7d832538e/volumes" Sep 17 15:42:33.468096 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:42:33.468066 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7800-9b635a\" is forbidden: User \"system:node:ip-10-0-128-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-15.ec2.internal' and this object" podUID="5d76685d-0f0e-44d2-ba96-09f7d832538e" pod="opendatahub/test-curl-7800-9b635a" Sep 17 15:42:33.558741 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.558708 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:33.750435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:33.750351 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:34.186678 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:34.186646 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" Sep 17 15:42:34.928311 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:34.928277 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:35.018196 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:35.018161 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:35.280246 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:35.280214 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:36.574587 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:36.574557 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:36.728287 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:36.728256 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:37.175960 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.175899 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:42:37.314975 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.314926 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:37.453630 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.453539 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:37.534192 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.534158 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5"] Sep 17 15:42:37.534569 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.534552 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d76685d-0f0e-44d2-ba96-09f7d832538e" containerName="test-curl-7800-9b635a" Sep 17 15:42:37.534654 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.534572 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d76685d-0f0e-44d2-ba96-09f7d832538e" containerName="test-curl-7800-9b635a" Sep 17 15:42:37.534654 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.534651 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d76685d-0f0e-44d2-ba96-09f7d832538e" containerName="test-curl-7800-9b635a" Sep 17 15:42:37.538708 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.538687 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" Sep 17 15:42:37.541258 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.541237 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-849b71-dockercfg-j6gpn\"" Sep 17 15:42:37.541346 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.541248 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-849b71\"" Sep 17 15:42:37.550393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.550372 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5"] Sep 17 15:42:37.687253 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.687213 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0537ab3-dc97-4bd8-adec-f6b1ccdda18e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5\" (UID: \"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" Sep 17 15:42:37.687655 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.687270 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4twgk\" (UniqueName: \"kubernetes.io/projected/e0537ab3-dc97-4bd8-adec-f6b1ccdda18e-kube-api-access-4twgk\") pod \"payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5\" (UID: \"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" Sep 17 15:42:37.787806 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.787770 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0537ab3-dc97-4bd8-adec-f6b1ccdda18e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5\" (UID: \"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" Sep 17 15:42:37.787983 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.787822 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4twgk\" (UniqueName: \"kubernetes.io/projected/e0537ab3-dc97-4bd8-adec-f6b1ccdda18e-kube-api-access-4twgk\") pod \"payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5\" (UID: \"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" Sep 17 15:42:37.788596 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.788576 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0537ab3-dc97-4bd8-adec-f6b1ccdda18e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5\" (UID: \"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" Sep 17 15:42:37.798777 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.798748 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4twgk\" (UniqueName: \"kubernetes.io/projected/e0537ab3-dc97-4bd8-adec-f6b1ccdda18e-kube-api-access-4twgk\") pod \"payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5\" (UID: \"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" Sep 17 15:42:37.848611 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.848582 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" Sep 17 15:42:37.891479 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.891449 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs"] Sep 17 15:42:37.896047 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.896021 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:37.900331 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.900286 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-28eba6-serving-cert\"" Sep 17 15:42:37.912706 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.912681 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs"] Sep 17 15:42:37.936762 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.936507 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:37.990316 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.990286 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtt9r\" (UniqueName: \"kubernetes.io/projected/22c532d1-779d-4796-baa3-ddf8ee933c3c-kube-api-access-gtt9r\") pod \"maas-api-e2e-autores-28eba6-86b997d447-r8hxs\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:37.990475 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.990328 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c532d1-779d-4796-baa3-ddf8ee933c3c-maas-api-tls\") pod \"maas-api-e2e-autores-28eba6-86b997d447-r8hxs\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:37.990475 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.990350 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c532d1-779d-4796-baa3-ddf8ee933c3c-openshift-service-ca\") pod \"maas-api-e2e-autores-28eba6-86b997d447-r8hxs\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:37.990475 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:37.990455 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c532d1-779d-4796-baa3-ddf8ee933c3c-metrics-certs\") pod \"maas-api-e2e-autores-28eba6-86b997d447-r8hxs\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:38.019987 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.019945 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:38.091569 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.091486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtt9r\" (UniqueName: \"kubernetes.io/projected/22c532d1-779d-4796-baa3-ddf8ee933c3c-kube-api-access-gtt9r\") pod \"maas-api-e2e-autores-28eba6-86b997d447-r8hxs\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:38.091569 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.091532 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c532d1-779d-4796-baa3-ddf8ee933c3c-maas-api-tls\") pod \"maas-api-e2e-autores-28eba6-86b997d447-r8hxs\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:38.091569 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.091553 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c532d1-779d-4796-baa3-ddf8ee933c3c-openshift-service-ca\") pod \"maas-api-e2e-autores-28eba6-86b997d447-r8hxs\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:38.091834 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.091608 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c532d1-779d-4796-baa3-ddf8ee933c3c-metrics-certs\") pod \"maas-api-e2e-autores-28eba6-86b997d447-r8hxs\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:38.092327 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.092288 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c532d1-779d-4796-baa3-ddf8ee933c3c-openshift-service-ca\") pod \"maas-api-e2e-autores-28eba6-86b997d447-r8hxs\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:38.094052 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.094029 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c532d1-779d-4796-baa3-ddf8ee933c3c-metrics-certs\") pod \"maas-api-e2e-autores-28eba6-86b997d447-r8hxs\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:38.094168 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.094118 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c532d1-779d-4796-baa3-ddf8ee933c3c-maas-api-tls\") pod \"maas-api-e2e-autores-28eba6-86b997d447-r8hxs\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:38.101717 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.101693 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtt9r\" (UniqueName: \"kubernetes.io/projected/22c532d1-779d-4796-baa3-ddf8ee933c3c-kube-api-access-gtt9r\") pod \"maas-api-e2e-autores-28eba6-86b997d447-r8hxs\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:38.126370 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.126348 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5"] Sep 17 15:42:38.128523 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:42:38.128501 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0537ab3_dc97_4bd8_adec_f6b1ccdda18e.slice/crio-5ab78939f7d279d04a2a47a1f8a0f31cfb674b8be6d32f4fe53944f3257af3af WatchSource:0}: Error finding container 5ab78939f7d279d04a2a47a1f8a0f31cfb674b8be6d32f4fe53944f3257af3af: Status 404 returned error can't find the container with id 5ab78939f7d279d04a2a47a1f8a0f31cfb674b8be6d32f4fe53944f3257af3af Sep 17 15:42:38.208498 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.208464 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" event={"ID":"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e","Type":"ContainerStarted","Data":"13406c072373bbfc622d7b7c108ea3a63005acb6056109b80fb45b23d9a26df4"} Sep 17 15:42:38.208616 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.208528 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" event={"ID":"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e","Type":"ContainerStarted","Data":"5ab78939f7d279d04a2a47a1f8a0f31cfb674b8be6d32f4fe53944f3257af3af"} Sep 17 15:42:38.208679 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.208654 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" Sep 17 15:42:38.214212 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.214191 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:38.229490 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.229445 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" podStartSLOduration=1.229428492 podStartE2EDuration="1.229428492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:38.227150038 +0000 UTC m=+745.223386794" watchObservedRunningTime="2026-09-17 15:42:38.229428492 +0000 UTC m=+745.225665238" Sep 17 15:42:38.510394 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.510362 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs"] Sep 17 15:42:38.510527 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:42:38.510502 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c532d1_779d_4796_baa3_ddf8ee933c3c.slice/crio-6010fa1aaceebacccddbeb6f45fab6c60aa52183a8d1587543d6998553db7204 WatchSource:0}: Error finding container 6010fa1aaceebacccddbeb6f45fab6c60aa52183a8d1587543d6998553db7204: Status 404 returned error can't find the container with id 6010fa1aaceebacccddbeb6f45fab6c60aa52183a8d1587543d6998553db7204 Sep 17 15:42:38.532900 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.532874 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:38.648250 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.648221 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:38.830111 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.830076 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq"] Sep 17 15:42:38.833741 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.833721 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" Sep 17 15:42:38.836723 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.836700 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-28eba6-dockercfg-x7lmb\"" Sep 17 15:42:38.836811 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.836738 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-28eba6\"" Sep 17 15:42:38.856319 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:38.856291 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq"] Sep 17 15:42:39.000247 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.000205 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ebe4b4-b059-44d9-931d-97ba6431f91d-plugins-config-volume\") pod \"payload-processing-e2e-autores-28eba6-bcd474885-2szsq\" (UID: \"24ebe4b4-b059-44d9-931d-97ba6431f91d\") " pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" Sep 17 15:42:39.000393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.000263 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ttzc\" (UniqueName: \"kubernetes.io/projected/24ebe4b4-b059-44d9-931d-97ba6431f91d-kube-api-access-7ttzc\") pod \"payload-processing-e2e-autores-28eba6-bcd474885-2szsq\" (UID: \"24ebe4b4-b059-44d9-931d-97ba6431f91d\") " pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" Sep 17 15:42:39.101624 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.101532 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ttzc\" (UniqueName: \"kubernetes.io/projected/24ebe4b4-b059-44d9-931d-97ba6431f91d-kube-api-access-7ttzc\") pod \"payload-processing-e2e-autores-28eba6-bcd474885-2szsq\" (UID: \"24ebe4b4-b059-44d9-931d-97ba6431f91d\") " pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" Sep 17 15:42:39.101754 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.101635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ebe4b4-b059-44d9-931d-97ba6431f91d-plugins-config-volume\") pod \"payload-processing-e2e-autores-28eba6-bcd474885-2szsq\" (UID: \"24ebe4b4-b059-44d9-931d-97ba6431f91d\") " pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" Sep 17 15:42:39.102358 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.102338 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ebe4b4-b059-44d9-931d-97ba6431f91d-plugins-config-volume\") pod \"payload-processing-e2e-autores-28eba6-bcd474885-2szsq\" (UID: \"24ebe4b4-b059-44d9-931d-97ba6431f91d\") " pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" Sep 17 15:42:39.114517 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.114493 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ttzc\" (UniqueName: \"kubernetes.io/projected/24ebe4b4-b059-44d9-931d-97ba6431f91d-kube-api-access-7ttzc\") pod \"payload-processing-e2e-autores-28eba6-bcd474885-2szsq\" (UID: \"24ebe4b4-b059-44d9-931d-97ba6431f91d\") " pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" Sep 17 15:42:39.144097 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.144070 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" Sep 17 15:42:39.182449 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.182409 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:39.216631 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.216600 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" event={"ID":"22c532d1-779d-4796-baa3-ddf8ee933c3c","Type":"ContainerStarted","Data":"589b1fb2437cfe3049dfb4231606ff092eee971557a479d5b936a3a2e38039f1"} Sep 17 15:42:39.216836 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.216697 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" event={"ID":"22c532d1-779d-4796-baa3-ddf8ee933c3c","Type":"ContainerStarted","Data":"6010fa1aaceebacccddbeb6f45fab6c60aa52183a8d1587543d6998553db7204"} Sep 17 15:42:39.217234 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.217217 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:39.245936 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.245858 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" podStartSLOduration=1.901924664 podStartE2EDuration="2.245842742s" totalImagesPullingTime="343.918078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:39.244472214 +0000 UTC m=+746.240708957" watchObservedRunningTime="2026-09-17 15:42:39.245842742 +0000 UTC m=+746.242079488" Sep 17 15:42:39.453429 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:42:39.453400 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ebe4b4_b059_44d9_931d_97ba6431f91d.slice/crio-43ae0ebf0db81c2c58db44dabf715ea41c85da5f800bdaaa361919876573d54a WatchSource:0}: Error finding container 43ae0ebf0db81c2c58db44dabf715ea41c85da5f800bdaaa361919876573d54a: Status 404 returned error can't find the container with id 43ae0ebf0db81c2c58db44dabf715ea41c85da5f800bdaaa361919876573d54a Sep 17 15:42:39.462544 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.462453 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq"] Sep 17 15:42:39.753506 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:39.753422 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:40.126994 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:40.126958 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:40.223803 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:40.223759 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" event={"ID":"24ebe4b4-b059-44d9-931d-97ba6431f91d","Type":"ContainerStarted","Data":"1b4a6d4741beb8087502422572bf7204cdeb772681e65db71ef7f65971d3e1d1"} Sep 17 15:42:40.224011 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:40.223823 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" event={"ID":"24ebe4b4-b059-44d9-931d-97ba6431f91d","Type":"ContainerStarted","Data":"43ae0ebf0db81c2c58db44dabf715ea41c85da5f800bdaaa361919876573d54a"} Sep 17 15:42:40.224011 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:40.223966 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" Sep 17 15:42:40.245591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:40.245540 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" podStartSLOduration=2.245515693 podStartE2EDuration="2.245515693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:40.244219399 +0000 UTC m=+747.240456145" watchObservedRunningTime="2026-09-17 15:42:40.245515693 +0000 UTC m=+747.241752440" Sep 17 15:42:40.396708 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:40.396626 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:40.708819 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:40.708732 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:40.856934 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:40.856876 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:41.333604 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:41.333568 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:41.498458 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:41.498426 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:41.579785 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:41.579751 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:41.991220 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:41.991184 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:42.788779 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:42.788734 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:43.809325 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:43.809296 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:44.189410 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:44.189338 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" Sep 17 15:42:44.241762 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:44.241725 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:44.352323 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:44.352292 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:44.456618 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:44.456538 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:44.676619 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:44.676591 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:44.924374 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:44.924344 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:45.054128 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:45.054096 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:45.186110 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:45.186032 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:45.193730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:45.193705 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" Sep 17 15:42:45.352657 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:45.352618 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:45.470864 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:45.470785 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:46.235907 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:46.235881 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:42:46.456757 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:46.456726 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:47.517696 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:47.517665 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:47.736553 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:47.736525 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:48.132975 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:48.132943 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:48.589993 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:48.589965 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:48.705320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:48.705287 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:48.820776 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:48.820745 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:49.218889 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.218856 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" Sep 17 15:42:49.581719 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.581689 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:49.661275 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.661245 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll"] Sep 17 15:42:49.664898 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.664877 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:49.667492 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.667473 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a2dc23-serving-cert\"" Sep 17 15:42:49.685220 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.685191 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll"] Sep 17 15:42:49.690562 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.690528 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d65jt\" (UniqueName: \"kubernetes.io/projected/57f6762f-ad32-458d-88b3-34ccb6232990-kube-api-access-d65jt\") pod \"maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:49.690830 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.690801 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f6762f-ad32-458d-88b3-34ccb6232990-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:49.690955 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.690855 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57f6762f-ad32-458d-88b3-34ccb6232990-maas-api-tls\") pod \"maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:49.690955 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.690893 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f6762f-ad32-458d-88b3-34ccb6232990-metrics-certs\") pod \"maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:49.698680 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.698650 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:49.791440 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.791396 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f6762f-ad32-458d-88b3-34ccb6232990-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:49.791613 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.791448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57f6762f-ad32-458d-88b3-34ccb6232990-maas-api-tls\") pod \"maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:49.791613 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.791481 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f6762f-ad32-458d-88b3-34ccb6232990-metrics-certs\") pod \"maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:49.791613 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.791522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d65jt\" (UniqueName: \"kubernetes.io/projected/57f6762f-ad32-458d-88b3-34ccb6232990-kube-api-access-d65jt\") pod \"maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:49.792202 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.792175 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f6762f-ad32-458d-88b3-34ccb6232990-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:49.794025 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.794000 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57f6762f-ad32-458d-88b3-34ccb6232990-maas-api-tls\") pod \"maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:49.794159 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.794064 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f6762f-ad32-458d-88b3-34ccb6232990-metrics-certs\") pod \"maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:49.803666 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.803640 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d65jt\" (UniqueName: \"kubernetes.io/projected/57f6762f-ad32-458d-88b3-34ccb6232990-kube-api-access-d65jt\") pod \"maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:49.836051 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.835974 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:49.949966 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.949898 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:49.976061 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:49.976026 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:50.100685 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.100608 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:50.267720 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.267687 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll"] Sep 17 15:42:50.268594 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:42:50.268566 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f6762f_ad32_458d_88b3_34ccb6232990.slice/crio-a45daa66704bb273843aa11a359f96fc87afe28bec6fb3d6ec44ff2cba5770de WatchSource:0}: Error finding container a45daa66704bb273843aa11a359f96fc87afe28bec6fb3d6ec44ff2cba5770de: Status 404 returned error can't find the container with id a45daa66704bb273843aa11a359f96fc87afe28bec6fb3d6ec44ff2cba5770de Sep 17 15:42:50.285132 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.285106 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" event={"ID":"57f6762f-ad32-458d-88b3-34ccb6232990","Type":"ContainerStarted","Data":"a45daa66704bb273843aa11a359f96fc87afe28bec6fb3d6ec44ff2cba5770de"} Sep 17 15:42:50.564142 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.564097 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:50.659486 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.659452 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:50.734199 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.734174 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8"] Sep 17 15:42:50.737867 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.737845 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" Sep 17 15:42:50.740302 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.740280 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a2dc23-dockercfg-rgnhq\"" Sep 17 15:42:50.740393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.740317 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a2dc23\"" Sep 17 15:42:50.749897 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.749870 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8"] Sep 17 15:42:50.804635 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.804553 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568f21c4-ab00-4891-9b22-22dd66d0575d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8\" (UID: \"568f21c4-ab00-4891-9b22-22dd66d0575d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" Sep 17 15:42:50.804807 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.804685 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k95tc\" (UniqueName: \"kubernetes.io/projected/568f21c4-ab00-4891-9b22-22dd66d0575d-kube-api-access-k95tc\") pod \"payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8\" (UID: \"568f21c4-ab00-4891-9b22-22dd66d0575d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" Sep 17 15:42:50.850816 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.850743 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7"] Sep 17 15:42:50.854288 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.854267 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" Sep 17 15:42:50.870785 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.870758 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7"] Sep 17 15:42:50.905407 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.905377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sknks\" (UniqueName: \"kubernetes.io/projected/c62ab279-57e6-429d-8adc-403d62adafa1-kube-api-access-sknks\") pod \"payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7\" (UID: \"c62ab279-57e6-429d-8adc-403d62adafa1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" Sep 17 15:42:50.905559 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.905507 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568f21c4-ab00-4891-9b22-22dd66d0575d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8\" (UID: \"568f21c4-ab00-4891-9b22-22dd66d0575d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" Sep 17 15:42:50.905559 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.905546 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62ab279-57e6-429d-8adc-403d62adafa1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7\" (UID: \"c62ab279-57e6-429d-8adc-403d62adafa1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" Sep 17 15:42:50.905684 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.905589 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k95tc\" (UniqueName: \"kubernetes.io/projected/568f21c4-ab00-4891-9b22-22dd66d0575d-kube-api-access-k95tc\") pod \"payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8\" (UID: \"568f21c4-ab00-4891-9b22-22dd66d0575d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" Sep 17 15:42:50.906325 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.906306 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568f21c4-ab00-4891-9b22-22dd66d0575d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8\" (UID: \"568f21c4-ab00-4891-9b22-22dd66d0575d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" Sep 17 15:42:50.929821 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:50.929794 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k95tc\" (UniqueName: \"kubernetes.io/projected/568f21c4-ab00-4891-9b22-22dd66d0575d-kube-api-access-k95tc\") pod \"payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8\" (UID: \"568f21c4-ab00-4891-9b22-22dd66d0575d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" Sep 17 15:42:51.007092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:51.007055 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62ab279-57e6-429d-8adc-403d62adafa1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7\" (UID: \"c62ab279-57e6-429d-8adc-403d62adafa1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" Sep 17 15:42:51.007251 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:51.007147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sknks\" (UniqueName: \"kubernetes.io/projected/c62ab279-57e6-429d-8adc-403d62adafa1-kube-api-access-sknks\") pod \"payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7\" (UID: \"c62ab279-57e6-429d-8adc-403d62adafa1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" Sep 17 15:42:51.007760 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:51.007739 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62ab279-57e6-429d-8adc-403d62adafa1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7\" (UID: \"c62ab279-57e6-429d-8adc-403d62adafa1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" Sep 17 15:42:51.020084 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:51.020052 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sknks\" (UniqueName: \"kubernetes.io/projected/c62ab279-57e6-429d-8adc-403d62adafa1-kube-api-access-sknks\") pod \"payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7\" (UID: \"c62ab279-57e6-429d-8adc-403d62adafa1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" Sep 17 15:42:51.059711 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:51.059674 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" Sep 17 15:42:51.119114 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:51.118937 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:51.165021 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:51.164991 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" Sep 17 15:42:51.232943 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:51.232093 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" Sep 17 15:42:51.293469 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:51.293429 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" event={"ID":"57f6762f-ad32-458d-88b3-34ccb6232990","Type":"ContainerStarted","Data":"bc5082140c8433d3f0b6b2e8333f6d802dc07714f7ccb2499cb8d5c0f3897853"} Sep 17 15:42:51.293660 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:51.293505 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:51.322393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:51.322330 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" podStartSLOduration=1.868317497 podStartE2EDuration="2.322303477s" totalImagesPullingTime="453.98598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:51.317758219 +0000 UTC m=+758.313994967" watchObservedRunningTime="2026-09-17 15:42:51.322303477 +0000 UTC m=+758.318540223" Sep 17 15:42:51.355781 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:51.355732 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8"] Sep 17 15:42:51.361085 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:42:51.361044 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568f21c4_ab00_4891_9b22_22dd66d0575d.slice/crio-0591753026e176be8711271b4a827d9f29aae44364d6bc5e49e55e4829d0df0b WatchSource:0}: Error finding container 0591753026e176be8711271b4a827d9f29aae44364d6bc5e49e55e4829d0df0b: Status 404 returned error can't find the container with id 0591753026e176be8711271b4a827d9f29aae44364d6bc5e49e55e4829d0df0b Sep 17 15:42:51.470942 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:42:51.469843 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62ab279_57e6_429d_8adc_403d62adafa1.slice/crio-72d7d3b999fc9e5ce053bfb5bea4ccbee19943fa429d6d07059203f5bea7df07 WatchSource:0}: Error finding container 72d7d3b999fc9e5ce053bfb5bea4ccbee19943fa429d6d07059203f5bea7df07: Status 404 returned error can't find the container with id 72d7d3b999fc9e5ce053bfb5bea4ccbee19943fa429d6d07059203f5bea7df07 Sep 17 15:42:51.474936 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:51.471759 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7"] Sep 17 15:42:52.304339 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:52.304298 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" event={"ID":"c62ab279-57e6-429d-8adc-403d62adafa1","Type":"ContainerStarted","Data":"b1cb6911644a0270d780efbdaf7f795dcfa8d2283cb4df6b80d56bb2c42d4749"} Sep 17 15:42:52.304928 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:52.304872 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" event={"ID":"c62ab279-57e6-429d-8adc-403d62adafa1","Type":"ContainerStarted","Data":"72d7d3b999fc9e5ce053bfb5bea4ccbee19943fa429d6d07059203f5bea7df07"} Sep 17 15:42:52.305399 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:52.305381 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" Sep 17 15:42:52.307453 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:52.307429 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" event={"ID":"568f21c4-ab00-4891-9b22-22dd66d0575d","Type":"ContainerStarted","Data":"1f03b5793c3fef22cfd3bc327257604a02364b46aac23f91d376890f28e67f6e"} Sep 17 15:42:52.307853 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:52.307835 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" Sep 17 15:42:52.308026 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:52.308010 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" event={"ID":"568f21c4-ab00-4891-9b22-22dd66d0575d","Type":"ContainerStarted","Data":"0591753026e176be8711271b4a827d9f29aae44364d6bc5e49e55e4829d0df0b"} Sep 17 15:42:52.340145 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:52.340100 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" podStartSLOduration=2.340087702 podStartE2EDuration="2.340087702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:52.338702176 +0000 UTC m=+759.334938921" watchObservedRunningTime="2026-09-17 15:42:52.340087702 +0000 UTC m=+759.336324482" Sep 17 15:42:52.366620 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:52.366573 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" podStartSLOduration=2.366561242 podStartE2EDuration="2.366561242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:52.364223927 +0000 UTC m=+759.360460673" watchObservedRunningTime="2026-09-17 15:42:52.366561242 +0000 UTC m=+759.362797988" Sep 17 15:42:52.463603 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:52.463569 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:53.717549 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:53.717511 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:54.505347 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:54.505308 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:55.527400 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:55.527356 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:55.627382 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:55.627334 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:56.389135 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:56.389097 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:56.803861 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:56.803825 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:56.902703 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:56.902668 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:57.166448 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:57.166362 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:57.313954 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:57.313929 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:42:57.990211 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:57.990180 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:58.162533 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:58.162501 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:59.031624 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:59.031592 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:42:59.897870 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:42:59.897839 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:00.780652 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:00.780620 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:01.361235 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:01.361205 2581 generic.go:356] "Generic (PLEG): container finished" podID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerID="99c8b3876b11686a643bb7e1e086d91921c1364a53f09abd066149b206809734" exitCode=0 Sep 17 15:43:01.361404 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:01.361287 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" event={"ID":"948331da-4b2c-4f6e-bf94-24551fdf86b5","Type":"ContainerDied","Data":"99c8b3876b11686a643bb7e1e086d91921c1364a53f09abd066149b206809734"} Sep 17 15:43:01.361673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:01.361659 2581 scope.go:117] "RemoveContainer" containerID="99c8b3876b11686a643bb7e1e086d91921c1364a53f09abd066149b206809734" Sep 17 15:43:01.362759 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:01.362735 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:43:01.362829 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:01.362796 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:43:01.362829 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:01.362826 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:43:01.522258 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:01.522217 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:02.170458 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:02.170422 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:43:02.374638 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:02.374599 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" event={"ID":"948331da-4b2c-4f6e-bf94-24551fdf86b5","Type":"ContainerStarted","Data":"4666d88018638d9ddbb2b0b89dc5d999b441951387d8b86e7c58c20b74779d08"} Sep 17 15:43:02.600877 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:02.600836 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:02.915110 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:02.915019 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:43:02.916189 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:02.916162 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 17 15:43:02.916307 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:02.916217 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" podUID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 17 15:43:03.473474 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:03.473440 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:03.914789 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:03.914751 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 17 15:43:03.914971 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:03.914813 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" podUID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 17 15:43:04.320834 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:04.320802 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" Sep 17 15:43:04.321055 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:04.320900 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" Sep 17 15:43:04.745667 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:04.745571 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:04.915476 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:04.915442 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 17 15:43:04.915639 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:04.915499 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" podUID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 17 15:43:05.915086 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:05.915002 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 17 15:43:05.915086 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:05.915061 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" podUID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 17 15:43:05.972247 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:05.972201 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:06.759621 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:06.759591 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:06.915572 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:06.915536 2581 patch_prober.go:28] interesting pod/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 17 15:43:06.915990 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:06.915597 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" podUID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 17 15:43:07.635738 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:07.635689 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:07.981429 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:07.981323 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:43:07.981869 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:07.981823 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:43:07.983042 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:07.983020 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:43:08.705972 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:08.705906 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:09.113535 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:09.113504 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:09.601209 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:09.601181 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:10.498745 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:10.498694 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:10.945637 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:10.945607 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh"] Sep 17 15:43:10.948087 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:10.948071 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:10.950322 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:10.950300 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"openshift-service-ca.crt\"" Sep 17 15:43:10.951220 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:10.951197 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"default-dockercfg-287z2\"" Sep 17 15:43:10.951305 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:10.951218 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"simulator-w4-c86e9a-kserve-self-signed-certs\"" Sep 17 15:43:10.951305 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:10.951256 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"kube-root-ca.crt\"" Sep 17 15:43:10.959541 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:10.959519 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh"] Sep 17 15:43:11.089539 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.089511 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-model-cache\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.089694 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.089550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-tls-certs\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.089694 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.089633 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-dshm\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.089694 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.089661 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-home\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.089694 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.089683 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77s9s\" (UniqueName: \"kubernetes.io/projected/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-kube-api-access-77s9s\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.089837 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.089710 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-tmp-dir\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.190169 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.190135 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-dshm\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.190306 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.190179 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-home\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.190348 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.190298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77s9s\" (UniqueName: \"kubernetes.io/projected/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-kube-api-access-77s9s\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.190348 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.190340 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-tmp-dir\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.190439 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.190389 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-model-cache\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.190439 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.190418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-tls-certs\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.190600 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.190580 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-home\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.190755 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.190736 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-tmp-dir\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.190831 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.190814 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-model-cache\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.192425 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.192385 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-dshm\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.192669 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.192652 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-tls-certs\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:11.921193 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:11.921160 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77s9s\" (UniqueName: \"kubernetes.io/projected/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-kube-api-access-77s9s\") pod \"simulator-w4-c86e9a-kserve-66fc66865c-txmlh\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:12.157736 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:12.157699 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:18.387438 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:18.387398 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh"] Sep 17 15:43:18.410210 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:43:18.410182 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod799214bd_9f3f_46a1_b3c5_b2a2b54a4d51.slice/crio-42e8102bb4b6ba3c59c988df392dd0c003f682940a669aa14c1294d8dfaf85f0 WatchSource:0}: Error finding container 42e8102bb4b6ba3c59c988df392dd0c003f682940a669aa14c1294d8dfaf85f0: Status 404 returned error can't find the container with id 42e8102bb4b6ba3c59c988df392dd0c003f682940a669aa14c1294d8dfaf85f0 Sep 17 15:43:18.481594 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:18.481568 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" event={"ID":"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51","Type":"ContainerStarted","Data":"e4809dacf664772296cde72dd45162ab7296e755e1c0b4e5f2ec8c277e7721e5"} Sep 17 15:43:18.481689 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:18.481606 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" event={"ID":"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51","Type":"ContainerStarted","Data":"42e8102bb4b6ba3c59c988df392dd0c003f682940a669aa14c1294d8dfaf85f0"} Sep 17 15:43:22.158320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:22.158270 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:22.158320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:22.158334 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:22.171025 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:22.170998 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:22.512200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:22.512117 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:43:24.484962 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:24.484893 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" podStartSLOduration=14.48487673 podStartE2EDuration="14.48487673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:43:24.483967436 +0000 UTC m=+791.480204184" watchObservedRunningTime="2026-09-17 15:43:24.48487673 +0000 UTC m=+791.481113476" Sep 17 15:43:29.785687 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:29.785652 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:35.215239 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:35.215204 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:35.329279 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:35.329248 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:36.104690 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:36.104603 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:36.775797 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:36.775762 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:36.866205 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:36.866172 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:36.968240 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:36.968201 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:37.045567 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:37.045489 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:37.198585 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:37.198556 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:37.681604 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:37.681572 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:37.879488 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:37.879435 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:38.014894 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:38.014800 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:38.255408 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:38.255378 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:38.352056 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:38.352025 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:38.453609 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:38.453577 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:39.139832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:39.139802 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:39.855193 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:39.855164 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:40.632470 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:40.632436 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:40.989817 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:40.989739 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:41.090486 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:41.090456 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:41.507409 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:41.507380 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:42.411508 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:42.411474 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:42.514390 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:42.514357 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:42.614176 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:42.614140 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:43.080011 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:43.079966 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:43.175750 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:43.175720 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:43.605842 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:43.605810 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:44.454560 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:44.454530 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:45.396750 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:45.396717 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:46.129137 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:46.129107 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:46.955792 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:46.955758 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:50.199555 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.199521 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws"] Sep 17 15:43:50.203233 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.203217 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.208113 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.208093 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"default-dockercfg-k96cw\"" Sep 17 15:43:50.208759 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.208699 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"openshift-service-ca.crt\"" Sep 17 15:43:50.208759 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.208699 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"simulator-w0-849b71-kserve-self-signed-certs\"" Sep 17 15:43:50.208985 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.208840 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"kube-root-ca.crt\"" Sep 17 15:43:50.231731 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.231697 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws"] Sep 17 15:43:50.233694 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.233669 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-tmp-dir\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.233820 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.233699 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z684w\" (UniqueName: \"kubernetes.io/projected/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-kube-api-access-z684w\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.233820 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.233742 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-home\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.233820 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.233794 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-dshm\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.234002 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.233849 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-tls-certs\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.234002 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.233874 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-model-cache\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.307901 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.307869 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:43:50.335519 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.335489 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-home\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.335759 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.335739 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-dshm\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.335948 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.335930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-tls-certs\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.336069 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.336053 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-model-cache\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.336209 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.336195 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-tmp-dir\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.336306 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.336292 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z684w\" (UniqueName: \"kubernetes.io/projected/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-kube-api-access-z684w\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.336458 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.336303 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-model-cache\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.336558 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.336535 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-tmp-dir\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.336623 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.336060 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-home\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.339011 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.338989 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-dshm\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.343196 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.343147 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-tls-certs\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.346848 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.346820 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z684w\" (UniqueName: \"kubernetes.io/projected/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-kube-api-access-z684w\") pod \"simulator-w0-849b71-kserve-f564bdc76-sbsws\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.512435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.512342 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:43:50.788579 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:50.788556 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws"] Sep 17 15:43:50.789908 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:43:50.789883 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8dab90a_6fbd_47aa_9c6a_04bf4ad2b073.slice/crio-8d1bb12c15200e641d575e342f4fe4d776d14a89e99e8c968cbc43796116c1f3 WatchSource:0}: Error finding container 8d1bb12c15200e641d575e342f4fe4d776d14a89e99e8c968cbc43796116c1f3: Status 404 returned error can't find the container with id 8d1bb12c15200e641d575e342f4fe4d776d14a89e99e8c968cbc43796116c1f3 Sep 17 15:43:51.662241 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:51.662201 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" event={"ID":"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073","Type":"ContainerStarted","Data":"104ac2092d0c94a68daddf7e43dfb6007ec55cca436d9f5704d812df9dbd2bba"} Sep 17 15:43:51.662241 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:51.662254 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" event={"ID":"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073","Type":"ContainerStarted","Data":"8d1bb12c15200e641d575e342f4fe4d776d14a89e99e8c968cbc43796116c1f3"} Sep 17 15:43:51.680744 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:43:51.680692 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" podStartSLOduration=1.680673512 podStartE2EDuration="1.680673512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:43:51.679779579 +0000 UTC m=+818.676016324" watchObservedRunningTime="2026-09-17 15:43:51.680673512 +0000 UTC m=+818.676910260" Sep 17 15:44:00.513237 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:00.513201 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:44:00.513237 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:00.513247 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:44:00.526239 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:00.526209 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:44:00.722375 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:00.722342 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:44:02.257399 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.257367 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v"] Sep 17 15:44:02.260873 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.260855 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.263279 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.263264 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-28eba6\"/\"kube-root-ca.crt\"" Sep 17 15:44:02.264032 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.264008 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-28eba6\"/\"openshift-service-ca.crt\"" Sep 17 15:44:02.264140 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.264099 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-28eba6\"/\"default-dockercfg-bxmzj\"" Sep 17 15:44:02.264208 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.264168 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-28eba6\"/\"autores-model-28eba6-kserve-self-signed-certs\"" Sep 17 15:44:02.276427 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.276395 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v"] Sep 17 15:44:02.288112 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.288082 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:02.341532 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.341493 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-home\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.341532 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.341534 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-dshm\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.341735 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.341549 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-tmp-dir\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.341735 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.341682 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5t8v\" (UniqueName: \"kubernetes.io/projected/accbe0f0-f65b-4902-970a-f1b46309803f-kube-api-access-c5t8v\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.341735 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.341721 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accbe0f0-f65b-4902-970a-f1b46309803f-tls-certs\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.341858 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.341802 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-model-cache\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.442482 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.442445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-model-cache\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.442670 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.442494 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-home\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.442670 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.442518 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-dshm\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.442670 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.442542 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-tmp-dir\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.442670 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.442595 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5t8v\" (UniqueName: \"kubernetes.io/projected/accbe0f0-f65b-4902-970a-f1b46309803f-kube-api-access-c5t8v\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.442670 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.442625 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accbe0f0-f65b-4902-970a-f1b46309803f-tls-certs\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.442988 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.442962 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-model-cache\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.443039 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.443001 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-home\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.443039 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.443026 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-tmp-dir\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.444801 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.444780 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-dshm\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.445083 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.445064 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accbe0f0-f65b-4902-970a-f1b46309803f-tls-certs\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.455778 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.455746 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5t8v\" (UniqueName: \"kubernetes.io/projected/accbe0f0-f65b-4902-970a-f1b46309803f-kube-api-access-c5t8v\") pod \"autores-model-28eba6-kserve-5fcf49746c-crx5v\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.570823 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.570794 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:02.844607 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:02.844578 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v"] Sep 17 15:44:02.846394 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:44:02.846355 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaccbe0f0_f65b_4902_970a_f1b46309803f.slice/crio-588dc31bd6ebe7bebace891fc2d82e186b47e6fed8987961b7683ffa61ef2096 WatchSource:0}: Error finding container 588dc31bd6ebe7bebace891fc2d82e186b47e6fed8987961b7683ffa61ef2096: Status 404 returned error can't find the container with id 588dc31bd6ebe7bebace891fc2d82e186b47e6fed8987961b7683ffa61ef2096 Sep 17 15:44:03.728489 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:03.728452 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" event={"ID":"accbe0f0-f65b-4902-970a-f1b46309803f","Type":"ContainerStarted","Data":"881dbfdddd9ecaa328c73752273607d9626df93bc713a8a992421ebad08076e6"} Sep 17 15:44:03.728864 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:03.728498 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" event={"ID":"accbe0f0-f65b-4902-970a-f1b46309803f","Type":"ContainerStarted","Data":"588dc31bd6ebe7bebace891fc2d82e186b47e6fed8987961b7683ffa61ef2096"} Sep 17 15:44:04.783235 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:04.783204 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:05.803901 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:05.803866 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:06.819431 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:06.819398 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:07.673096 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:07.673068 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:08.483555 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:08.483523 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:09.279441 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:09.279411 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:09.960432 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:09.960394 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:10.811481 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:10.811442 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:11.509040 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:11.509008 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:12.312505 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:12.312478 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:12.408164 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:12.408130 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:12.513236 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:12.513199 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:12.571342 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:12.571263 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:12.571342 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:12.571322 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:12.583467 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:12.583435 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:12.611548 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:12.611520 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:12.611694 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:12.611658 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" podStartSLOduration=10.611643082 podStartE2EDuration="10.611643082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:44:03.749309032 +0000 UTC m=+830.745545790" watchObservedRunningTime="2026-09-17 15:44:12.611643082 +0000 UTC m=+839.607879827" Sep 17 15:44:12.696495 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:12.696446 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:12.781644 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:12.781602 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:12.792908 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:12.792882 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:44:13.230111 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:13.230078 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:13.527144 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:13.527110 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:13.622467 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:13.622436 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:13.725704 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:13.725671 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:14.585089 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:14.585051 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:15.391087 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:15.391059 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:16.192798 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:16.192748 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:16.281096 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:16.281065 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:17.232619 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:17.232585 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:18.065489 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:18.065460 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:18.381204 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:18.381123 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:19.208699 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:19.208665 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:19.310675 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:19.310644 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:19.464380 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:19.464299 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:19.793288 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:19.793262 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:19.892666 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:19.892633 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:20.378896 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:20.378861 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:21.206148 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:21.206114 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:21.969803 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:21.969771 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:36.467102 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:36.467035 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:37.772380 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:37.772347 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:38.542315 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:38.542282 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:39.504837 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:39.504809 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:40.303239 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:40.303210 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:41.081771 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:41.081738 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:41.895177 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:41.895142 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:42.678884 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:42.678852 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:43.399513 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:43.399480 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:44.321647 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:44.321616 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:45.078012 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:45.077981 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:45.810374 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:45.810345 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:46.924908 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:46.924874 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:47.734758 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:47.734724 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v"] Sep 17 15:44:47.843087 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:47.843056 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:48.343236 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:48.343194 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:49.022758 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.022727 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs"] Sep 17 15:44:49.023021 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.022996 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" podUID="22c532d1-779d-4796-baa3-ddf8ee933c3c" containerName="maas-api" containerID="cri-o://589b1fb2437cfe3049dfb4231606ff092eee971557a479d5b936a3a2e38039f1" gracePeriod=30 Sep 17 15:44:49.076595 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.076567 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:49.121832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.121798 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq"] Sep 17 15:44:49.122106 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.122078 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" podUID="24ebe4b4-b059-44d9-931d-97ba6431f91d" containerName="payload-processing" containerID="cri-o://1b4a6d4741beb8087502422572bf7204cdeb772681e65db71ef7f65971d3e1d1" gracePeriod=30 Sep 17 15:44:49.166369 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.166339 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:49.266936 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.266875 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:49.291889 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.291867 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:44:49.360776 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.360744 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c532d1-779d-4796-baa3-ddf8ee933c3c-metrics-certs\") pod \"22c532d1-779d-4796-baa3-ddf8ee933c3c\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " Sep 17 15:44:49.361273 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.360848 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtt9r\" (UniqueName: \"kubernetes.io/projected/22c532d1-779d-4796-baa3-ddf8ee933c3c-kube-api-access-gtt9r\") pod \"22c532d1-779d-4796-baa3-ddf8ee933c3c\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " Sep 17 15:44:49.361273 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.360886 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c532d1-779d-4796-baa3-ddf8ee933c3c-maas-api-tls\") pod \"22c532d1-779d-4796-baa3-ddf8ee933c3c\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " Sep 17 15:44:49.361273 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.360990 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c532d1-779d-4796-baa3-ddf8ee933c3c-openshift-service-ca\") pod \"22c532d1-779d-4796-baa3-ddf8ee933c3c\" (UID: \"22c532d1-779d-4796-baa3-ddf8ee933c3c\") " Sep 17 15:44:49.361451 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.361424 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c532d1-779d-4796-baa3-ddf8ee933c3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "22c532d1-779d-4796-baa3-ddf8ee933c3c" (UID: "22c532d1-779d-4796-baa3-ddf8ee933c3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:44:49.363617 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.363589 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c532d1-779d-4796-baa3-ddf8ee933c3c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "22c532d1-779d-4796-baa3-ddf8ee933c3c" (UID: "22c532d1-779d-4796-baa3-ddf8ee933c3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:44:49.363723 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.363662 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c532d1-779d-4796-baa3-ddf8ee933c3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "22c532d1-779d-4796-baa3-ddf8ee933c3c" (UID: "22c532d1-779d-4796-baa3-ddf8ee933c3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:44:49.363801 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.363732 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c532d1-779d-4796-baa3-ddf8ee933c3c-kube-api-access-gtt9r" (OuterVolumeSpecName: "kube-api-access-gtt9r") pod "22c532d1-779d-4796-baa3-ddf8ee933c3c" (UID: "22c532d1-779d-4796-baa3-ddf8ee933c3c"). InnerVolumeSpecName "kube-api-access-gtt9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:44:49.366825 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.366801 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:49.382333 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.382312 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" Sep 17 15:44:49.462573 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.462536 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ebe4b4-b059-44d9-931d-97ba6431f91d-plugins-config-volume\") pod \"24ebe4b4-b059-44d9-931d-97ba6431f91d\" (UID: \"24ebe4b4-b059-44d9-931d-97ba6431f91d\") " Sep 17 15:44:49.462729 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.462633 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ttzc\" (UniqueName: \"kubernetes.io/projected/24ebe4b4-b059-44d9-931d-97ba6431f91d-kube-api-access-7ttzc\") pod \"24ebe4b4-b059-44d9-931d-97ba6431f91d\" (UID: \"24ebe4b4-b059-44d9-931d-97ba6431f91d\") " Sep 17 15:44:49.462890 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.462871 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c532d1-779d-4796-baa3-ddf8ee933c3c-metrics-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:44:49.462978 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.462895 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtt9r\" (UniqueName: \"kubernetes.io/projected/22c532d1-779d-4796-baa3-ddf8ee933c3c-kube-api-access-gtt9r\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:44:49.462978 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.462891 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ebe4b4-b059-44d9-931d-97ba6431f91d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24ebe4b4-b059-44d9-931d-97ba6431f91d" (UID: "24ebe4b4-b059-44d9-931d-97ba6431f91d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:44:49.462978 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.462925 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c532d1-779d-4796-baa3-ddf8ee933c3c-maas-api-tls\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:44:49.462978 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.462961 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c532d1-779d-4796-baa3-ddf8ee933c3c-openshift-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:44:49.464952 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.464929 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ebe4b4-b059-44d9-931d-97ba6431f91d-kube-api-access-7ttzc" (OuterVolumeSpecName: "kube-api-access-7ttzc") pod "24ebe4b4-b059-44d9-931d-97ba6431f91d" (UID: "24ebe4b4-b059-44d9-931d-97ba6431f91d"). InnerVolumeSpecName "kube-api-access-7ttzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:44:49.467200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.467177 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:49.563870 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.563837 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ebe4b4-b059-44d9-931d-97ba6431f91d-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:44:49.563870 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.563869 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ttzc\" (UniqueName: \"kubernetes.io/projected/24ebe4b4-b059-44d9-931d-97ba6431f91d-kube-api-access-7ttzc\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:44:49.583223 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.583199 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:49.837507 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.837423 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:49.991332 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.991300 2581 generic.go:356] "Generic (PLEG): container finished" podID="24ebe4b4-b059-44d9-931d-97ba6431f91d" containerID="1b4a6d4741beb8087502422572bf7204cdeb772681e65db71ef7f65971d3e1d1" exitCode=0 Sep 17 15:44:49.991526 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.991350 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" Sep 17 15:44:49.991526 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.991382 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" event={"ID":"24ebe4b4-b059-44d9-931d-97ba6431f91d","Type":"ContainerDied","Data":"1b4a6d4741beb8087502422572bf7204cdeb772681e65db71ef7f65971d3e1d1"} Sep 17 15:44:49.991526 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.991437 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq" event={"ID":"24ebe4b4-b059-44d9-931d-97ba6431f91d","Type":"ContainerDied","Data":"43ae0ebf0db81c2c58db44dabf715ea41c85da5f800bdaaa361919876573d54a"} Sep 17 15:44:49.991526 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.991462 2581 scope.go:117] "RemoveContainer" containerID="1b4a6d4741beb8087502422572bf7204cdeb772681e65db71ef7f65971d3e1d1" Sep 17 15:44:49.992898 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.992869 2581 generic.go:356] "Generic (PLEG): container finished" podID="22c532d1-779d-4796-baa3-ddf8ee933c3c" containerID="589b1fb2437cfe3049dfb4231606ff092eee971557a479d5b936a3a2e38039f1" exitCode=0 Sep 17 15:44:49.993036 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.992907 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" Sep 17 15:44:49.993036 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.992950 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" event={"ID":"22c532d1-779d-4796-baa3-ddf8ee933c3c","Type":"ContainerDied","Data":"589b1fb2437cfe3049dfb4231606ff092eee971557a479d5b936a3a2e38039f1"} Sep 17 15:44:49.993036 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:49.993032 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs" event={"ID":"22c532d1-779d-4796-baa3-ddf8ee933c3c","Type":"ContainerDied","Data":"6010fa1aaceebacccddbeb6f45fab6c60aa52183a8d1587543d6998553db7204"} Sep 17 15:44:50.001467 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:50.001451 2581 scope.go:117] "RemoveContainer" containerID="1b4a6d4741beb8087502422572bf7204cdeb772681e65db71ef7f65971d3e1d1" Sep 17 15:44:50.001766 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:44:50.001741 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4a6d4741beb8087502422572bf7204cdeb772681e65db71ef7f65971d3e1d1\": container with ID starting with 1b4a6d4741beb8087502422572bf7204cdeb772681e65db71ef7f65971d3e1d1 not found: ID does not exist" containerID="1b4a6d4741beb8087502422572bf7204cdeb772681e65db71ef7f65971d3e1d1" Sep 17 15:44:50.001830 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:50.001769 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4a6d4741beb8087502422572bf7204cdeb772681e65db71ef7f65971d3e1d1"} err="failed to get container status \"1b4a6d4741beb8087502422572bf7204cdeb772681e65db71ef7f65971d3e1d1\": rpc error: code = NotFound desc = could not find container \"1b4a6d4741beb8087502422572bf7204cdeb772681e65db71ef7f65971d3e1d1\": container with ID starting with 1b4a6d4741beb8087502422572bf7204cdeb772681e65db71ef7f65971d3e1d1 not found: ID does not exist" Sep 17 15:44:50.001830 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:50.001786 2581 scope.go:117] "RemoveContainer" containerID="589b1fb2437cfe3049dfb4231606ff092eee971557a479d5b936a3a2e38039f1" Sep 17 15:44:50.009605 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:50.009588 2581 scope.go:117] "RemoveContainer" containerID="589b1fb2437cfe3049dfb4231606ff092eee971557a479d5b936a3a2e38039f1" Sep 17 15:44:50.009858 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:44:50.009842 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"589b1fb2437cfe3049dfb4231606ff092eee971557a479d5b936a3a2e38039f1\": container with ID starting with 589b1fb2437cfe3049dfb4231606ff092eee971557a479d5b936a3a2e38039f1 not found: ID does not exist" containerID="589b1fb2437cfe3049dfb4231606ff092eee971557a479d5b936a3a2e38039f1" Sep 17 15:44:50.009908 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:50.009867 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"589b1fb2437cfe3049dfb4231606ff092eee971557a479d5b936a3a2e38039f1"} err="failed to get container status \"589b1fb2437cfe3049dfb4231606ff092eee971557a479d5b936a3a2e38039f1\": rpc error: code = NotFound desc = could not find container \"589b1fb2437cfe3049dfb4231606ff092eee971557a479d5b936a3a2e38039f1\": container with ID starting with 589b1fb2437cfe3049dfb4231606ff092eee971557a479d5b936a3a2e38039f1 not found: ID does not exist" Sep 17 15:44:50.013691 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:50.013670 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs"] Sep 17 15:44:50.016877 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:50.016858 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-28eba6-86b997d447-r8hxs"] Sep 17 15:44:50.029020 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:50.028990 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq"] Sep 17 15:44:50.030993 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:50.030976 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-28eba6-bcd474885-2szsq"] Sep 17 15:44:50.396779 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:50.396746 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:51.436964 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:51.436904 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:51.462187 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:51.462158 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c532d1-779d-4796-baa3-ddf8ee933c3c" path="/var/lib/kubelet/pods/22c532d1-779d-4796-baa3-ddf8ee933c3c/volumes" Sep 17 15:44:51.462522 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:51.462509 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ebe4b4-b059-44d9-931d-97ba6431f91d" path="/var/lib/kubelet/pods/24ebe4b4-b059-44d9-931d-97ba6431f91d/volumes" Sep 17 15:44:52.223636 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:52.223606 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:53.136570 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:53.136538 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:53.988970 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:53.988934 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:54.715381 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:54.715349 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:54.916715 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:54.916682 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:58.142754 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:58.142717 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:58.221374 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:58.221337 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:58.322250 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:58.322221 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:58.418858 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:58.418780 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:58.678626 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:58.678548 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:59.564119 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:59.564081 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:59.659818 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:59.659784 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:44:59.991314 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:44:59.991239 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:00.183921 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.183876 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k"] Sep 17 15:45:00.184289 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.184274 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c532d1-779d-4796-baa3-ddf8ee933c3c" containerName="maas-api" Sep 17 15:45:00.184333 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.184290 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c532d1-779d-4796-baa3-ddf8ee933c3c" containerName="maas-api" Sep 17 15:45:00.184333 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.184307 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ebe4b4-b059-44d9-931d-97ba6431f91d" containerName="payload-processing" Sep 17 15:45:00.184333 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.184316 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ebe4b4-b059-44d9-931d-97ba6431f91d" containerName="payload-processing" Sep 17 15:45:00.184445 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.184383 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ebe4b4-b059-44d9-931d-97ba6431f91d" containerName="payload-processing" Sep 17 15:45:00.184445 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.184391 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c532d1-779d-4796-baa3-ddf8ee933c3c" containerName="maas-api" Sep 17 15:45:00.189121 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.189087 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k" Sep 17 15:45:00.191770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.191745 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\"" Sep 17 15:45:00.202384 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.202356 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k"] Sep 17 15:45:00.224873 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.224845 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k" Sep 17 15:45:00.369482 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.369450 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:00.510034 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.510012 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k"] Sep 17 15:45:00.512456 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:45:00.512427 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e3c420_d4c3_4556_9dae_7e14144a28fc.slice/crio-00611063eca718cb0d3280ad6bf32037e177fcc9f354d40e21bb2962cadcec38 WatchSource:0}: Error finding container 00611063eca718cb0d3280ad6bf32037e177fcc9f354d40e21bb2962cadcec38: Status 404 returned error can't find the container with id 00611063eca718cb0d3280ad6bf32037e177fcc9f354d40e21bb2962cadcec38 Sep 17 15:45:00.523540 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.523512 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:00.603301 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:00.603220 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:01.063063 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:01.063025 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k" event={"ID":"75e3c420-d4c3-4556-9dae-7e14144a28fc","Type":"ContainerStarted","Data":"00611063eca718cb0d3280ad6bf32037e177fcc9f354d40e21bb2962cadcec38"} Sep 17 15:45:01.817896 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:01.817857 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:02.549260 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:02.549223 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:02.723991 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:02.723955 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:02.753664 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:02.753612 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" podUID="accbe0f0-f65b-4902-970a-f1b46309803f" containerName="main" containerID="cri-o://881dbfdddd9ecaa328c73752273607d9626df93bc713a8a992421ebad08076e6" gracePeriod=45 Sep 17 15:45:02.782662 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:02.782611 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" podUID="accbe0f0-f65b-4902-970a-f1b46309803f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Sep 17 15:45:02.819495 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:02.819412 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:03.225792 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.225766 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:45:03.288226 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.288196 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-dshm\") pod \"accbe0f0-f65b-4902-970a-f1b46309803f\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " Sep 17 15:45:03.288410 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.288267 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-model-cache\") pod \"accbe0f0-f65b-4902-970a-f1b46309803f\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " Sep 17 15:45:03.288410 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.288289 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-tmp-dir\") pod \"accbe0f0-f65b-4902-970a-f1b46309803f\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " Sep 17 15:45:03.288410 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.288336 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accbe0f0-f65b-4902-970a-f1b46309803f-tls-certs\") pod \"accbe0f0-f65b-4902-970a-f1b46309803f\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " Sep 17 15:45:03.288410 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.288356 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5t8v\" (UniqueName: \"kubernetes.io/projected/accbe0f0-f65b-4902-970a-f1b46309803f-kube-api-access-c5t8v\") pod \"accbe0f0-f65b-4902-970a-f1b46309803f\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " Sep 17 15:45:03.288640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.288415 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-home\") pod \"accbe0f0-f65b-4902-970a-f1b46309803f\" (UID: \"accbe0f0-f65b-4902-970a-f1b46309803f\") " Sep 17 15:45:03.288640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.288541 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "accbe0f0-f65b-4902-970a-f1b46309803f" (UID: "accbe0f0-f65b-4902-970a-f1b46309803f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:45:03.288640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.288553 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-model-cache" (OuterVolumeSpecName: "model-cache") pod "accbe0f0-f65b-4902-970a-f1b46309803f" (UID: "accbe0f0-f65b-4902-970a-f1b46309803f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:45:03.288785 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.288658 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-model-cache\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:45:03.288785 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.288677 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-tmp-dir\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:45:03.288785 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.288737 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-home" (OuterVolumeSpecName: "home") pod "accbe0f0-f65b-4902-970a-f1b46309803f" (UID: "accbe0f0-f65b-4902-970a-f1b46309803f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:45:03.290489 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.290465 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-dshm" (OuterVolumeSpecName: "dshm") pod "accbe0f0-f65b-4902-970a-f1b46309803f" (UID: "accbe0f0-f65b-4902-970a-f1b46309803f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:45:03.290594 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.290489 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accbe0f0-f65b-4902-970a-f1b46309803f-kube-api-access-c5t8v" (OuterVolumeSpecName: "kube-api-access-c5t8v") pod "accbe0f0-f65b-4902-970a-f1b46309803f" (UID: "accbe0f0-f65b-4902-970a-f1b46309803f"). InnerVolumeSpecName "kube-api-access-c5t8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:45:03.290594 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.290524 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/accbe0f0-f65b-4902-970a-f1b46309803f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "accbe0f0-f65b-4902-970a-f1b46309803f" (UID: "accbe0f0-f65b-4902-970a-f1b46309803f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:45:03.389563 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.389488 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5t8v\" (UniqueName: \"kubernetes.io/projected/accbe0f0-f65b-4902-970a-f1b46309803f-kube-api-access-c5t8v\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:45:03.389563 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.389519 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-home\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:45:03.389563 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.389529 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accbe0f0-f65b-4902-970a-f1b46309803f-dshm\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:45:03.389563 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.389537 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accbe0f0-f65b-4902-970a-f1b46309803f-tls-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:45:03.773891 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:03.773807 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:04.081474 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:04.081443 2581 generic.go:356] "Generic (PLEG): container finished" podID="accbe0f0-f65b-4902-970a-f1b46309803f" containerID="881dbfdddd9ecaa328c73752273607d9626df93bc713a8a992421ebad08076e6" exitCode=0 Sep 17 15:45:04.081952 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:04.081493 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" Sep 17 15:45:04.081952 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:04.081526 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" event={"ID":"accbe0f0-f65b-4902-970a-f1b46309803f","Type":"ContainerDied","Data":"881dbfdddd9ecaa328c73752273607d9626df93bc713a8a992421ebad08076e6"} Sep 17 15:45:04.081952 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:04.081567 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v" event={"ID":"accbe0f0-f65b-4902-970a-f1b46309803f","Type":"ContainerDied","Data":"588dc31bd6ebe7bebace891fc2d82e186b47e6fed8987961b7683ffa61ef2096"} Sep 17 15:45:04.081952 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:04.081598 2581 scope.go:117] "RemoveContainer" containerID="881dbfdddd9ecaa328c73752273607d9626df93bc713a8a992421ebad08076e6" Sep 17 15:45:04.083515 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:04.083490 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k" event={"ID":"75e3c420-d4c3-4556-9dae-7e14144a28fc","Type":"ContainerStarted","Data":"8f0f1c74bf841ca73780d7220016e231e1243cc9c00967ade7ca61f78680b357"} Sep 17 15:45:04.090754 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:04.090738 2581 scope.go:117] "RemoveContainer" containerID="881dbfdddd9ecaa328c73752273607d9626df93bc713a8a992421ebad08076e6" Sep 17 15:45:04.091039 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:45:04.091015 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"881dbfdddd9ecaa328c73752273607d9626df93bc713a8a992421ebad08076e6\": container with ID starting with 881dbfdddd9ecaa328c73752273607d9626df93bc713a8a992421ebad08076e6 not found: ID does not exist" containerID="881dbfdddd9ecaa328c73752273607d9626df93bc713a8a992421ebad08076e6" Sep 17 15:45:04.091152 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:04.091053 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"881dbfdddd9ecaa328c73752273607d9626df93bc713a8a992421ebad08076e6"} err="failed to get container status \"881dbfdddd9ecaa328c73752273607d9626df93bc713a8a992421ebad08076e6\": rpc error: code = NotFound desc = could not find container \"881dbfdddd9ecaa328c73752273607d9626df93bc713a8a992421ebad08076e6\": container with ID starting with 881dbfdddd9ecaa328c73752273607d9626df93bc713a8a992421ebad08076e6 not found: ID does not exist" Sep 17 15:45:04.101182 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:04.101162 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v"] Sep 17 15:45:04.106400 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:04.106382 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-28eba6/autores-model-28eba6-kserve-5fcf49746c-crx5v"] Sep 17 15:45:04.119923 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:04.119880 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k" podStartSLOduration=1.494937554 podStartE2EDuration="4.11986767s" totalImagesPullingTime="2.624930116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:04.119348918 +0000 UTC m=+891.115585664" watchObservedRunningTime="2026-09-17 15:45:04.11986767 +0000 UTC m=+891.116104460" Sep 17 15:45:05.091468 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:05.091421 2581 generic.go:356] "Generic (PLEG): container finished" podID="75e3c420-d4c3-4556-9dae-7e14144a28fc" containerID="8f0f1c74bf841ca73780d7220016e231e1243cc9c00967ade7ca61f78680b357" exitCode=0 Sep 17 15:45:05.091836 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:05.091498 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k" event={"ID":"75e3c420-d4c3-4556-9dae-7e14144a28fc","Type":"ContainerDied","Data":"8f0f1c74bf841ca73780d7220016e231e1243cc9c00967ade7ca61f78680b357"} Sep 17 15:45:05.467367 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:05.467273 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="accbe0f0-f65b-4902-970a-f1b46309803f" path="/var/lib/kubelet/pods/accbe0f0-f65b-4902-970a-f1b46309803f/volumes" Sep 17 15:45:05.617968 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:05.617876 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:06.228734 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:06.228707 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k" Sep 17 15:45:06.319778 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:06.319744 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:07.103504 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.103477 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k" Sep 17 15:45:07.103656 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.103508 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k" event={"ID":"75e3c420-d4c3-4556-9dae-7e14144a28fc","Type":"ContainerDied","Data":"00611063eca718cb0d3280ad6bf32037e177fcc9f354d40e21bb2962cadcec38"} Sep 17 15:45:07.103656 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.103547 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00611063eca718cb0d3280ad6bf32037e177fcc9f354d40e21bb2962cadcec38" Sep 17 15:45:07.365980 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.365889 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh"] Sep 17 15:45:07.366466 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.366450 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e3c420-d4c3-4556-9dae-7e14144a28fc" containerName="cleanup" Sep 17 15:45:07.366532 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.366470 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e3c420-d4c3-4556-9dae-7e14144a28fc" containerName="cleanup" Sep 17 15:45:07.366532 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.366502 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="accbe0f0-f65b-4902-970a-f1b46309803f" containerName="main" Sep 17 15:45:07.366532 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.366511 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="accbe0f0-f65b-4902-970a-f1b46309803f" containerName="main" Sep 17 15:45:07.366624 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.366618 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e3c420-d4c3-4556-9dae-7e14144a28fc" containerName="cleanup" Sep 17 15:45:07.366663 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.366635 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="accbe0f0-f65b-4902-970a-f1b46309803f" containerName="main" Sep 17 15:45:07.370224 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.370203 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.372494 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.372473 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-23da8b90-serving-cert\"" Sep 17 15:45:07.386459 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.386435 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh"] Sep 17 15:45:07.445134 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.445108 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:07.527223 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.527189 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l9cl\" (UniqueName: \"kubernetes.io/projected/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-kube-api-access-2l9cl\") pod \"maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.527393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.527293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-metrics-certs\") pod \"maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.527393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.527326 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-maas-api-tls\") pod \"maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.527480 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.527412 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-openshift-service-ca\") pod \"maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.537398 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.537371 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:07.625816 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.625745 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:07.627843 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.627817 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-openshift-service-ca\") pod \"maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.627961 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.627860 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l9cl\" (UniqueName: \"kubernetes.io/projected/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-kube-api-access-2l9cl\") pod \"maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.627961 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.627944 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-metrics-certs\") pod \"maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.628088 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.627969 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-maas-api-tls\") pod \"maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.628575 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.628557 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-openshift-service-ca\") pod \"maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.630381 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.630360 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-metrics-certs\") pod \"maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.630636 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.630614 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-maas-api-tls\") pod \"maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.638110 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.638090 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l9cl\" (UniqueName: \"kubernetes.io/projected/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-kube-api-access-2l9cl\") pod \"maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.681004 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.680976 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:07.963050 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:07.963023 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh"] Sep 17 15:45:07.964647 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:45:07.964618 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7a6395_d9b3_4dda_8937_4e887b12a8aa.slice/crio-5fdc57e6f99978aec7c68586dbc1f7b0503d7c761a9c9ab322994adc700f628a WatchSource:0}: Error finding container 5fdc57e6f99978aec7c68586dbc1f7b0503d7c761a9c9ab322994adc700f628a: Status 404 returned error can't find the container with id 5fdc57e6f99978aec7c68586dbc1f7b0503d7c761a9c9ab322994adc700f628a Sep 17 15:45:08.110815 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:08.110784 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" event={"ID":"2d7a6395-d9b3-4dda-8937-4e887b12a8aa","Type":"ContainerStarted","Data":"5fdc57e6f99978aec7c68586dbc1f7b0503d7c761a9c9ab322994adc700f628a"} Sep 17 15:45:08.119432 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:08.119408 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:08.354752 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:08.354722 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:09.117881 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:09.117839 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" event={"ID":"2d7a6395-d9b3-4dda-8937-4e887b12a8aa","Type":"ContainerStarted","Data":"f454e33baa2ee3e1a644191e733d347d974c4251b014e2a32c7ca3dc6843b1e8"} Sep 17 15:45:09.118393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:09.117941 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:09.141139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:09.141092 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" podStartSLOduration=1.734516276 podStartE2EDuration="2.141074804s" totalImagesPullingTime="406.558528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:09.138060876 +0000 UTC m=+896.134297622" watchObservedRunningTime="2026-09-17 15:45:09.141074804 +0000 UTC m=+896.137311550" Sep 17 15:45:15.129148 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:15.129115 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:20.092015 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:20.091973 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:20.612863 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:20.612826 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:20.716516 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:20.716484 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:29.718607 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:29.718573 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:30.389639 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:30.389603 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:31.463796 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:31.463767 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:32.089071 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:32.089038 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:32.938334 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:32.938304 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:33.614754 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:33.614717 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:33.800668 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:33.800635 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:33.950732 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:33.950646 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:34.345613 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:34.345585 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:34.638112 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:34.638014 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:34.732056 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:34.732026 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:34.870014 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:34.869985 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:35.658530 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:35.658498 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:36.580820 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:36.580785 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:37.433902 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:37.433867 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:37.556685 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:37.556653 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:37.782830 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:37.782807 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:38.031802 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:38.031769 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:38.135706 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:38.135625 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:38.876002 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:38.875966 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk"] Sep 17 15:45:38.883005 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:38.882979 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:38.885366 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:38.885343 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-5b08fa-serving-cert\"" Sep 17 15:45:38.896059 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:38.896030 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk"] Sep 17 15:45:38.925287 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:38.925258 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:38.996753 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:38.996720 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvw9t\" (UniqueName: \"kubernetes.io/projected/f9c570cc-c292-4388-9e6a-a52c0c43cc40-kube-api-access-gvw9t\") pod \"maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:38.996905 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:38.996781 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c570cc-c292-4388-9e6a-a52c0c43cc40-metrics-certs\") pod \"maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:38.996905 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:38.996867 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c570cc-c292-4388-9e6a-a52c0c43cc40-openshift-service-ca\") pod \"maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:38.997056 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:38.996940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c570cc-c292-4388-9e6a-a52c0c43cc40-maas-api-tls\") pod \"maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:39.026016 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.025990 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:39.097728 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.097699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c570cc-c292-4388-9e6a-a52c0c43cc40-openshift-service-ca\") pod \"maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:39.097883 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.097771 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c570cc-c292-4388-9e6a-a52c0c43cc40-maas-api-tls\") pod \"maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:39.097883 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.097809 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvw9t\" (UniqueName: \"kubernetes.io/projected/f9c570cc-c292-4388-9e6a-a52c0c43cc40-kube-api-access-gvw9t\") pod \"maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:39.097883 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.097843 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c570cc-c292-4388-9e6a-a52c0c43cc40-metrics-certs\") pod \"maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:39.098498 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.098470 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c570cc-c292-4388-9e6a-a52c0c43cc40-openshift-service-ca\") pod \"maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:39.100181 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.100161 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c570cc-c292-4388-9e6a-a52c0c43cc40-maas-api-tls\") pod \"maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:39.100270 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.100216 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c570cc-c292-4388-9e6a-a52c0c43cc40-metrics-certs\") pod \"maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:39.107085 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.107062 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvw9t\" (UniqueName: \"kubernetes.io/projected/f9c570cc-c292-4388-9e6a-a52c0c43cc40-kube-api-access-gvw9t\") pod \"maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:39.144785 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.144726 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:39.195657 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.195628 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:39.474629 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.474605 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk"] Sep 17 15:45:39.476107 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:45:39.476079 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c570cc_c292_4388_9e6a_a52c0c43cc40.slice/crio-ba1ac251de75170795ae0ac937f641846f484a3de3c0eabc86567644a39e0b34 WatchSource:0}: Error finding container ba1ac251de75170795ae0ac937f641846f484a3de3c0eabc86567644a39e0b34: Status 404 returned error can't find the container with id ba1ac251de75170795ae0ac937f641846f484a3de3c0eabc86567644a39e0b34 Sep 17 15:45:39.482017 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.481996 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:39.595974 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.595939 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:39.912333 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:39.912305 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:40.298452 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:40.298416 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" event={"ID":"f9c570cc-c292-4388-9e6a-a52c0c43cc40","Type":"ContainerStarted","Data":"f944f0acf1984466e650bf340a4df2d88a51def452f13af9e91bbed38809a912"} Sep 17 15:45:40.298703 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:40.298640 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:40.298703 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:40.298681 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" event={"ID":"f9c570cc-c292-4388-9e6a-a52c0c43cc40","Type":"ContainerStarted","Data":"ba1ac251de75170795ae0ac937f641846f484a3de3c0eabc86567644a39e0b34"} Sep 17 15:45:40.347454 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:40.347394 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" podStartSLOduration=1.975007503 podStartE2EDuration="2.347377497s" totalImagesPullingTime="372.369994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:40.343761389 +0000 UTC m=+927.339998134" watchObservedRunningTime="2026-09-17 15:45:40.347377497 +0000 UTC m=+927.343614244" Sep 17 15:45:41.511358 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:41.511324 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:41.739805 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:41.739768 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:42.506471 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:42.506434 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:43.284275 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:43.284239 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:44.079151 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:44.079117 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:44.855708 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:44.855669 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh"] Sep 17 15:45:44.856153 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:44.855860 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" podUID="2d7a6395-d9b3-4dda-8937-4e887b12a8aa" containerName="maas-api" containerID="cri-o://f454e33baa2ee3e1a644191e733d347d974c4251b014e2a32c7ca3dc6843b1e8" gracePeriod=30 Sep 17 15:45:44.888875 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:44.888846 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:44.985115 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:44.985086 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:45.069789 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.069756 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:45.103522 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.103486 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:45.151544 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.151465 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-maas-api-tls\") pod \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " Sep 17 15:45:45.151703 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.151571 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-openshift-service-ca\") pod \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " Sep 17 15:45:45.151703 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.151602 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l9cl\" (UniqueName: \"kubernetes.io/projected/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-kube-api-access-2l9cl\") pod \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " Sep 17 15:45:45.151703 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.151686 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-metrics-certs\") pod \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\" (UID: \"2d7a6395-d9b3-4dda-8937-4e887b12a8aa\") " Sep 17 15:45:45.152020 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.151984 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2d7a6395-d9b3-4dda-8937-4e887b12a8aa" (UID: "2d7a6395-d9b3-4dda-8937-4e887b12a8aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:45:45.153593 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.153565 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2d7a6395-d9b3-4dda-8937-4e887b12a8aa" (UID: "2d7a6395-d9b3-4dda-8937-4e887b12a8aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:45:45.153687 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.153643 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2d7a6395-d9b3-4dda-8937-4e887b12a8aa" (UID: "2d7a6395-d9b3-4dda-8937-4e887b12a8aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:45:45.153730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.153694 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-kube-api-access-2l9cl" (OuterVolumeSpecName: "kube-api-access-2l9cl") pod "2d7a6395-d9b3-4dda-8937-4e887b12a8aa" (UID: "2d7a6395-d9b3-4dda-8937-4e887b12a8aa"). InnerVolumeSpecName "kube-api-access-2l9cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:45:45.253332 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.253295 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-openshift-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:45:45.253332 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.253325 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l9cl\" (UniqueName: \"kubernetes.io/projected/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-kube-api-access-2l9cl\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:45:45.253332 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.253335 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-metrics-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:45:45.253563 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.253344 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d7a6395-d9b3-4dda-8937-4e887b12a8aa-maas-api-tls\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:45:45.327354 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.327312 2581 generic.go:356] "Generic (PLEG): container finished" podID="2d7a6395-d9b3-4dda-8937-4e887b12a8aa" containerID="f454e33baa2ee3e1a644191e733d347d974c4251b014e2a32c7ca3dc6843b1e8" exitCode=0 Sep 17 15:45:45.327354 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.327354 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" Sep 17 15:45:45.327557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.327374 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" event={"ID":"2d7a6395-d9b3-4dda-8937-4e887b12a8aa","Type":"ContainerDied","Data":"f454e33baa2ee3e1a644191e733d347d974c4251b014e2a32c7ca3dc6843b1e8"} Sep 17 15:45:45.327557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.327446 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh" event={"ID":"2d7a6395-d9b3-4dda-8937-4e887b12a8aa","Type":"ContainerDied","Data":"5fdc57e6f99978aec7c68586dbc1f7b0503d7c761a9c9ab322994adc700f628a"} Sep 17 15:45:45.327557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.327470 2581 scope.go:117] "RemoveContainer" containerID="f454e33baa2ee3e1a644191e733d347d974c4251b014e2a32c7ca3dc6843b1e8" Sep 17 15:45:45.337151 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.337125 2581 scope.go:117] "RemoveContainer" containerID="f454e33baa2ee3e1a644191e733d347d974c4251b014e2a32c7ca3dc6843b1e8" Sep 17 15:45:45.337490 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:45:45.337461 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f454e33baa2ee3e1a644191e733d347d974c4251b014e2a32c7ca3dc6843b1e8\": container with ID starting with f454e33baa2ee3e1a644191e733d347d974c4251b014e2a32c7ca3dc6843b1e8 not found: ID does not exist" containerID="f454e33baa2ee3e1a644191e733d347d974c4251b014e2a32c7ca3dc6843b1e8" Sep 17 15:45:45.337614 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.337501 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f454e33baa2ee3e1a644191e733d347d974c4251b014e2a32c7ca3dc6843b1e8"} err="failed to get container status \"f454e33baa2ee3e1a644191e733d347d974c4251b014e2a32c7ca3dc6843b1e8\": rpc error: code = NotFound desc = could not find container \"f454e33baa2ee3e1a644191e733d347d974c4251b014e2a32c7ca3dc6843b1e8\": container with ID starting with f454e33baa2ee3e1a644191e733d347d974c4251b014e2a32c7ca3dc6843b1e8 not found: ID does not exist" Sep 17 15:45:45.349310 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.349286 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh"] Sep 17 15:45:45.356138 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.356116 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-23da8b90-6c8f9fb74b-6q6xh"] Sep 17 15:45:45.462694 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.462615 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d7a6395-d9b3-4dda-8937-4e887b12a8aa" path="/var/lib/kubelet/pods/2d7a6395-d9b3-4dda-8937-4e887b12a8aa/volumes" Sep 17 15:45:45.768774 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.768696 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:45.841411 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:45.841376 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:46.027246 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:46.027212 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:46.120333 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:46.120299 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:46.309976 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:46.309892 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:45:47.833448 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:47.833417 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:48.705709 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:48.705674 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:49.368437 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:49.368409 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:49.860796 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:49.860760 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:50.580922 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:50.580872 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:51.561588 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:51.561555 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:56.013892 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:56.013863 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:56.116985 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:56.116945 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:56.233901 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:56.233870 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:56.332537 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:56.332490 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:56.811770 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:56.811706 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:45:59.907366 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:45:59.907302 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:01.370556 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:01.370525 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:02.141145 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:02.141114 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:02.276955 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:02.276897 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:02.371490 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:02.371458 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:03.170353 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:03.170326 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:03.514019 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:03.513903 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:04.106465 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:04.106408 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:04.899502 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:04.899418 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:05.765280 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:05.765250 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:15.971651 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:15.971614 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:16.079722 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:16.079642 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:22.732290 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:22.732253 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:29.193553 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:29.193513 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:29.313668 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:29.313618 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:29.440384 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:29.440353 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:39.166183 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:39.166078 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:39.328598 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:39.328554 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:45.364462 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:45.364429 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:51.789585 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:51.789531 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:51.903203 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:51.903166 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:46:52.035900 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:46:52.035864 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:00.196316 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:00.196276 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:02.545241 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:02.545190 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:03.394865 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:03.394826 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:05.700083 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:05.700037 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:08.947257 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:08.947228 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:09.959286 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:09.959254 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:10.772738 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:10.772707 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:11.037825 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:11.037795 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:11.170167 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:11.170116 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:11.822700 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:11.822664 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:11.931986 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:11.931935 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:12.033359 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:12.033318 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:12.607183 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:12.607153 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:13.771614 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:13.771578 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:13.892831 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:13.892783 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:14.805415 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:14.805361 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:15.831119 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:15.831080 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:20.249259 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.249223 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk"] Sep 17 15:47:20.249747 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.249444 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" podUID="f9c570cc-c292-4388-9e6a-a52c0c43cc40" containerName="maas-api" containerID="cri-o://f944f0acf1984466e650bf340a4df2d88a51def452f13af9e91bbed38809a912" gracePeriod=30 Sep 17 15:47:20.344046 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.344002 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:20.493724 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.493696 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:47:20.579908 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.579880 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c570cc-c292-4388-9e6a-a52c0c43cc40-metrics-certs\") pod \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " Sep 17 15:47:20.580205 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.579965 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvw9t\" (UniqueName: \"kubernetes.io/projected/f9c570cc-c292-4388-9e6a-a52c0c43cc40-kube-api-access-gvw9t\") pod \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " Sep 17 15:47:20.580205 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.580077 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c570cc-c292-4388-9e6a-a52c0c43cc40-openshift-service-ca\") pod \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " Sep 17 15:47:20.580205 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.580111 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c570cc-c292-4388-9e6a-a52c0c43cc40-maas-api-tls\") pod \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\" (UID: \"f9c570cc-c292-4388-9e6a-a52c0c43cc40\") " Sep 17 15:47:20.580492 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.580463 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c570cc-c292-4388-9e6a-a52c0c43cc40-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f9c570cc-c292-4388-9e6a-a52c0c43cc40" (UID: "f9c570cc-c292-4388-9e6a-a52c0c43cc40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:47:20.581990 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.581942 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c570cc-c292-4388-9e6a-a52c0c43cc40-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f9c570cc-c292-4388-9e6a-a52c0c43cc40" (UID: "f9c570cc-c292-4388-9e6a-a52c0c43cc40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:47:20.582085 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.582060 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c570cc-c292-4388-9e6a-a52c0c43cc40-kube-api-access-gvw9t" (OuterVolumeSpecName: "kube-api-access-gvw9t") pod "f9c570cc-c292-4388-9e6a-a52c0c43cc40" (UID: "f9c570cc-c292-4388-9e6a-a52c0c43cc40"). InnerVolumeSpecName "kube-api-access-gvw9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:47:20.582152 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.582133 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c570cc-c292-4388-9e6a-a52c0c43cc40-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f9c570cc-c292-4388-9e6a-a52c0c43cc40" (UID: "f9c570cc-c292-4388-9e6a-a52c0c43cc40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:47:20.598752 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.598726 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:20.681458 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.681430 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c570cc-c292-4388-9e6a-a52c0c43cc40-openshift-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:47:20.681458 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.681454 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9c570cc-c292-4388-9e6a-a52c0c43cc40-maas-api-tls\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:47:20.681458 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.681463 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c570cc-c292-4388-9e6a-a52c0c43cc40-metrics-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:47:20.681646 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.681472 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvw9t\" (UniqueName: \"kubernetes.io/projected/f9c570cc-c292-4388-9e6a-a52c0c43cc40-kube-api-access-gvw9t\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:47:20.837299 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.837216 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:20.841613 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.841586 2581 generic.go:356] "Generic (PLEG): container finished" podID="f9c570cc-c292-4388-9e6a-a52c0c43cc40" containerID="f944f0acf1984466e650bf340a4df2d88a51def452f13af9e91bbed38809a912" exitCode=0 Sep 17 15:47:20.841714 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.841629 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" Sep 17 15:47:20.841714 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.841671 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" event={"ID":"f9c570cc-c292-4388-9e6a-a52c0c43cc40","Type":"ContainerDied","Data":"f944f0acf1984466e650bf340a4df2d88a51def452f13af9e91bbed38809a912"} Sep 17 15:47:20.841801 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.841735 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk" event={"ID":"f9c570cc-c292-4388-9e6a-a52c0c43cc40","Type":"ContainerDied","Data":"ba1ac251de75170795ae0ac937f641846f484a3de3c0eabc86567644a39e0b34"} Sep 17 15:47:20.841801 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.841769 2581 scope.go:117] "RemoveContainer" containerID="f944f0acf1984466e650bf340a4df2d88a51def452f13af9e91bbed38809a912" Sep 17 15:47:20.852337 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.852320 2581 scope.go:117] "RemoveContainer" containerID="f944f0acf1984466e650bf340a4df2d88a51def452f13af9e91bbed38809a912" Sep 17 15:47:20.852669 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:47:20.852649 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f944f0acf1984466e650bf340a4df2d88a51def452f13af9e91bbed38809a912\": container with ID starting with f944f0acf1984466e650bf340a4df2d88a51def452f13af9e91bbed38809a912 not found: ID does not exist" containerID="f944f0acf1984466e650bf340a4df2d88a51def452f13af9e91bbed38809a912" Sep 17 15:47:20.852743 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.852678 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f944f0acf1984466e650bf340a4df2d88a51def452f13af9e91bbed38809a912"} err="failed to get container status \"f944f0acf1984466e650bf340a4df2d88a51def452f13af9e91bbed38809a912\": rpc error: code = NotFound desc = could not find container \"f944f0acf1984466e650bf340a4df2d88a51def452f13af9e91bbed38809a912\": container with ID starting with f944f0acf1984466e650bf340a4df2d88a51def452f13af9e91bbed38809a912 not found: ID does not exist" Sep 17 15:47:20.931303 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.931267 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk"] Sep 17 15:47:20.957072 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:20.957042 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5b08fa-5b7d4789fd-wjtvk"] Sep 17 15:47:21.461777 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:21.461746 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9c570cc-c292-4388-9e6a-a52c0c43cc40" path="/var/lib/kubelet/pods/f9c570cc-c292-4388-9e6a-a52c0c43cc40/volumes" Sep 17 15:47:21.565165 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:21.565133 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:21.932075 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:21.932038 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:22.383104 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:22.383075 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:23.958088 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:23.958051 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:24.164564 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:24.164533 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:24.403797 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:24.403765 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:24.695264 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:24.695182 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:25.518968 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:25.518934 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:26.505316 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:26.505286 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:27.330413 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:27.330381 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:29.510969 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:29.510932 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:29.609388 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:29.609357 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:29.774478 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:29.774398 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:30.339167 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.339129 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:30.549889 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.549857 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df"] Sep 17 15:47:30.550295 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.550255 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7a6395-d9b3-4dda-8937-4e887b12a8aa" containerName="maas-api" Sep 17 15:47:30.550295 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.550268 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7a6395-d9b3-4dda-8937-4e887b12a8aa" containerName="maas-api" Sep 17 15:47:30.550295 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.550289 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c570cc-c292-4388-9e6a-a52c0c43cc40" containerName="maas-api" Sep 17 15:47:30.550295 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.550295 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c570cc-c292-4388-9e6a-a52c0c43cc40" containerName="maas-api" Sep 17 15:47:30.550434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.550360 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9c570cc-c292-4388-9e6a-a52c0c43cc40" containerName="maas-api" Sep 17 15:47:30.550434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.550371 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7a6395-d9b3-4dda-8937-4e887b12a8aa" containerName="maas-api" Sep 17 15:47:30.554928 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.554891 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.559311 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.559291 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-d9afa3-openshift-default-dockercfg-kdknq\"" Sep 17 15:47:30.581455 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.581430 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df"] Sep 17 15:47:30.638245 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.638149 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:30.674283 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.674249 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8557711-d68e-4763-b8db-abef7b4d0452-istiod-ca-cert\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.674283 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.674284 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx586\" (UniqueName: \"kubernetes.io/projected/b8557711-d68e-4763-b8db-abef7b4d0452-kube-api-access-sx586\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.674468 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.674319 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-workload-socket\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.674468 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.674345 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-credential-socket\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.674468 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.674360 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8557711-d68e-4763-b8db-abef7b4d0452-istio-podinfo\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.674468 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.674419 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-istio-envoy\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.674468 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.674450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-istio-data\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.674643 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.674478 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-workload-certs\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.674643 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.674547 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8557711-d68e-4763-b8db-abef7b4d0452-istio-token\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.775444 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.775415 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-workload-socket\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.775606 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.775450 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-credential-socket\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.775606 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.775466 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8557711-d68e-4763-b8db-abef7b4d0452-istio-podinfo\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.775606 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.775492 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-istio-envoy\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.775606 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.775508 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-istio-data\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.775606 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.775537 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-workload-certs\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.775606 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.775593 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8557711-d68e-4763-b8db-abef7b4d0452-istio-token\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.776097 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.775645 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8557711-d68e-4763-b8db-abef7b4d0452-istiod-ca-cert\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.776097 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.775670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx586\" (UniqueName: \"kubernetes.io/projected/b8557711-d68e-4763-b8db-abef7b4d0452-kube-api-access-sx586\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.776097 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.775855 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-workload-socket\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.776097 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.776050 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-credential-socket\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.776317 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.776100 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-istio-data\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.776317 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.776282 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-workload-certs\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.776808 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.776784 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8557711-d68e-4763-b8db-abef7b4d0452-istiod-ca-cert\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.778263 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.778235 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-istio-envoy\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.778466 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.778451 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8557711-d68e-4763-b8db-abef7b4d0452-istio-podinfo\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.787591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.787565 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx586\" (UniqueName: \"kubernetes.io/projected/b8557711-d68e-4763-b8db-abef7b4d0452-kube-api-access-sx586\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.787837 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.787820 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8557711-d68e-4763-b8db-abef7b4d0452-istio-token\") pod \"e2e-noten-d9afa3-openshift-default-f8849555b-zg5df\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:30.854431 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.854385 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:30.865381 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:30.865355 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:31.160828 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:31.160797 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df"] Sep 17 15:47:31.162526 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:47:31.162488 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8557711_d68e_4763_b8db_abef7b4d0452.slice/crio-45777f80f72664b24d6136de49ed60b55f881de2694667b0e9f64550753cd8b0 WatchSource:0}: Error finding container 45777f80f72664b24d6136de49ed60b55f881de2694667b0e9f64550753cd8b0: Status 404 returned error can't find the container with id 45777f80f72664b24d6136de49ed60b55f881de2694667b0e9f64550753cd8b0 Sep 17 15:47:31.164705 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:31.164670 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:47:31.164796 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:31.164761 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:47:31.164865 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:31.164814 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:47:31.904026 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:31.903986 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" event={"ID":"b8557711-d68e-4763-b8db-abef7b4d0452","Type":"ContainerStarted","Data":"a7419c92d5612e69446eae046ecb642c9231337d1e90f46af879071a0d741872"} Sep 17 15:47:31.904461 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:31.904042 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" event={"ID":"b8557711-d68e-4763-b8db-abef7b4d0452","Type":"ContainerStarted","Data":"45777f80f72664b24d6136de49ed60b55f881de2694667b0e9f64550753cd8b0"} Sep 17 15:47:31.934500 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:31.934452 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" podStartSLOduration=1.934435445 podStartE2EDuration="1.934435445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:31.933904343 +0000 UTC m=+1038.930141091" watchObservedRunningTime="2026-09-17 15:47:31.934435445 +0000 UTC m=+1038.930672192" Sep 17 15:47:32.348371 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:32.348340 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:32.500599 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:32.500569 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:32.636168 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:32.636077 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:32.866079 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:32.866047 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:32.872244 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:32.872216 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:32.911290 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:32.911200 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:32.912222 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:32.912197 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:47:33.736309 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:33.736276 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:33.853836 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:33.853807 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:33.986814 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:33.986726 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:34.306985 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.306950 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6"] Sep 17 15:47:34.310436 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.310418 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.314445 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.314421 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-70207142-openshift-default-dockercfg-25gfh\"" Sep 17 15:47:34.330898 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.330873 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6"] Sep 17 15:47:34.409380 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.409346 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-istio-data\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.409591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.409403 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc97239-052f-43c6-8595-858206808cf3-istio-podinfo\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.409591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.409461 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc97239-052f-43c6-8595-858206808cf3-istio-token\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.409591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.409543 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-istio-envoy\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.409591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.409588 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f25dq\" (UniqueName: \"kubernetes.io/projected/edc97239-052f-43c6-8595-858206808cf3-kube-api-access-f25dq\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.409818 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.409617 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-credential-socket\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.409818 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.409649 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc97239-052f-43c6-8595-858206808cf3-istiod-ca-cert\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.409818 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.409740 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-workload-certs\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.409818 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.409796 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-workload-socket\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.468194 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.468160 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:34.511027 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.510995 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-istio-data\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.511198 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.511038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc97239-052f-43c6-8595-858206808cf3-istio-podinfo\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.511198 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.511065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc97239-052f-43c6-8595-858206808cf3-istio-token\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.511198 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.511101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-istio-envoy\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.511198 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.511137 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f25dq\" (UniqueName: \"kubernetes.io/projected/edc97239-052f-43c6-8595-858206808cf3-kube-api-access-f25dq\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.511485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.511205 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-credential-socket\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.511485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.511259 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc97239-052f-43c6-8595-858206808cf3-istiod-ca-cert\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.511485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.511334 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-workload-certs\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.511485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.511388 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-istio-data\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.511485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.511401 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-workload-socket\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.511750 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.511710 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-workload-socket\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.511808 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.511785 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-workload-certs\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.511864 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.511785 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-credential-socket\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.512133 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.512108 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc97239-052f-43c6-8595-858206808cf3-istiod-ca-cert\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.513351 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.513328 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-istio-envoy\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.513580 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.513565 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc97239-052f-43c6-8595-858206808cf3-istio-podinfo\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.550503 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.550470 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f25dq\" (UniqueName: \"kubernetes.io/projected/edc97239-052f-43c6-8595-858206808cf3-kube-api-access-f25dq\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.568508 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.568445 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc97239-052f-43c6-8595-858206808cf3-istio-token\") pod \"e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.621853 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.621821 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:34.772594 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.772566 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:34.930290 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.930231 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6"] Sep 17 15:47:34.932430 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:47:34.932404 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc97239_052f_43c6_8595_858206808cf3.slice/crio-3731de8d1b507a35f7a0880c52fbb45e857c7430c913504f2e247d1e9c50dd93 WatchSource:0}: Error finding container 3731de8d1b507a35f7a0880c52fbb45e857c7430c913504f2e247d1e9c50dd93: Status 404 returned error can't find the container with id 3731de8d1b507a35f7a0880c52fbb45e857c7430c913504f2e247d1e9c50dd93 Sep 17 15:47:34.934560 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.934529 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:47:34.934644 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.934593 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:47:34.934644 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:34.934619 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:47:35.929263 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:35.929233 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" event={"ID":"edc97239-052f-43c6-8595-858206808cf3","Type":"ContainerStarted","Data":"c99ef676aa921516e4aa3190cc5128703a64c192bc28a653d5e621e8656257d4"} Sep 17 15:47:35.929634 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:35.929280 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" event={"ID":"edc97239-052f-43c6-8595-858206808cf3","Type":"ContainerStarted","Data":"3731de8d1b507a35f7a0880c52fbb45e857c7430c913504f2e247d1e9c50dd93"} Sep 17 15:47:35.955617 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:35.955443 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" podStartSLOduration=1.955423518 podStartE2EDuration="1.955423518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:35.954302913 +0000 UTC m=+1042.950539672" watchObservedRunningTime="2026-09-17 15:47:35.955423518 +0000 UTC m=+1042.951660267" Sep 17 15:47:36.622011 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:36.621977 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:36.627258 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:36.627232 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:36.934903 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:36.934821 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:36.935698 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:36.935677 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:47:37.964107 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:37.964075 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:38.147259 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:38.147223 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:39.508497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:39.508460 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:40.716497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:40.716468 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:41.763638 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:41.763598 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:42.790201 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:42.790174 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:42.955307 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:42.955279 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:43.468490 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:43.468415 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:43.586523 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:43.586478 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:46.068539 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:46.068505 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:48.707925 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:48.707892 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:47:52.608478 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:47:52.608442 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:00.768414 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:00.768297 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:02.077959 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:02.077902 2581 generic.go:356] "Generic (PLEG): container finished" podID="edc97239-052f-43c6-8595-858206808cf3" containerID="c99ef676aa921516e4aa3190cc5128703a64c192bc28a653d5e621e8656257d4" exitCode=0 Sep 17 15:48:02.078363 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:02.077976 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" event={"ID":"edc97239-052f-43c6-8595-858206808cf3","Type":"ContainerDied","Data":"c99ef676aa921516e4aa3190cc5128703a64c192bc28a653d5e621e8656257d4"} Sep 17 15:48:02.078363 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:02.078357 2581 scope.go:117] "RemoveContainer" containerID="c99ef676aa921516e4aa3190cc5128703a64c192bc28a653d5e621e8656257d4" Sep 17 15:48:02.079406 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:02.079382 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:48:02.079460 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:02.079444 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:48:02.079493 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:02.079475 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:48:03.098216 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:03.098172 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" event={"ID":"edc97239-052f-43c6-8595-858206808cf3","Type":"ContainerStarted","Data":"be0de193e5d5056395de0ade88265dcda9e91af111413d33b6d613b8255edf2c"} Sep 17 15:48:03.622272 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:03.622230 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:48:03.623576 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:03.623549 2581 patch_prober.go:28] interesting pod/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 17 15:48:03.623699 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:03.623613 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" podUID="edc97239-052f-43c6-8595-858206808cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 17 15:48:04.024399 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:04.024307 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:04.622547 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:04.622510 2581 patch_prober.go:28] interesting pod/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 17 15:48:04.622956 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:04.622576 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" podUID="edc97239-052f-43c6-8595-858206808cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 17 15:48:05.622604 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:05.622553 2581 patch_prober.go:28] interesting pod/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 17 15:48:05.622997 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:05.622628 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" podUID="edc97239-052f-43c6-8595-858206808cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 17 15:48:05.715751 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:05.715718 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:06.531943 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:06.531894 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:06.622664 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:06.622609 2581 patch_prober.go:28] interesting pod/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 17 15:48:06.623078 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:06.622703 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" podUID="edc97239-052f-43c6-8595-858206808cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 17 15:48:06.935106 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:06.935019 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:48:07.402295 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:07.402264 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:07.622434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:07.622390 2581 patch_prober.go:28] interesting pod/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 17 15:48:07.622603 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:07.622454 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" podUID="edc97239-052f-43c6-8595-858206808cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 17 15:48:08.114277 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:08.114252 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:08.626251 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:08.626220 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:48:08.627177 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:08.627156 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:48:09.008878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:09.008799 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:09.215243 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:09.215209 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:09.801824 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:09.801785 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:10.180424 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:10.180332 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:11.349554 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:11.349520 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:16.448857 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:16.448820 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:16.646213 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:16.646180 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:18.106100 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:18.106065 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:19.224602 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:19.224570 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:20.140109 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:20.140065 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:21.270202 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:21.270170 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:22.562080 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:22.562045 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:23.719262 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:23.719235 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:27.710180 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:27.710144 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:27.837662 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:27.837628 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:27.945637 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:27.945603 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:28.043558 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:28.043527 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:28.455367 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:28.455288 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:28.573112 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:28.573083 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:29.482310 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:29.482278 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:30.321849 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:30.321816 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:31.393427 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:31.393391 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:32.329991 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:32.329949 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:35.140772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:35.140733 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:35.221531 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:35.221501 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:35.421394 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:35.421297 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6"] Sep 17 15:48:35.422252 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:35.421572 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" podUID="edc97239-052f-43c6-8595-858206808cf3" containerName="istio-proxy" containerID="cri-o://be0de193e5d5056395de0ade88265dcda9e91af111413d33b6d613b8255edf2c" gracePeriod=30 Sep 17 15:48:35.463946 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:35.463902 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:35.838099 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:35.838065 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:35.981188 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:35.981160 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:36.077374 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:36.077342 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:36.875531 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:36.875479 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:38.628756 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:38.628715 2581 patch_prober.go:28] interesting pod/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 15:48:38.629153 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:38.628779 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" podUID="edc97239-052f-43c6-8595-858206808cf3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 15:48:40.668441 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.668419 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:48:40.737765 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.737676 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc97239-052f-43c6-8595-858206808cf3-istio-token\") pod \"edc97239-052f-43c6-8595-858206808cf3\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " Sep 17 15:48:40.737765 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.737716 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-istio-data\") pod \"edc97239-052f-43c6-8595-858206808cf3\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " Sep 17 15:48:40.737765 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.737731 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-workload-certs\") pod \"edc97239-052f-43c6-8595-858206808cf3\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " Sep 17 15:48:40.737765 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.737753 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f25dq\" (UniqueName: \"kubernetes.io/projected/edc97239-052f-43c6-8595-858206808cf3-kube-api-access-f25dq\") pod \"edc97239-052f-43c6-8595-858206808cf3\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " Sep 17 15:48:40.738113 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.737807 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-credential-socket\") pod \"edc97239-052f-43c6-8595-858206808cf3\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " Sep 17 15:48:40.738113 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.737845 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-workload-socket\") pod \"edc97239-052f-43c6-8595-858206808cf3\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " Sep 17 15:48:40.738113 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.737902 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc97239-052f-43c6-8595-858206808cf3-istio-podinfo\") pod \"edc97239-052f-43c6-8595-858206808cf3\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " Sep 17 15:48:40.738113 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.738002 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc97239-052f-43c6-8595-858206808cf3-istiod-ca-cert\") pod \"edc97239-052f-43c6-8595-858206808cf3\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " Sep 17 15:48:40.738113 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.738048 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-istio-envoy\") pod \"edc97239-052f-43c6-8595-858206808cf3\" (UID: \"edc97239-052f-43c6-8595-858206808cf3\") " Sep 17 15:48:40.738113 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.738073 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-istio-data" (OuterVolumeSpecName: "istio-data") pod "edc97239-052f-43c6-8595-858206808cf3" (UID: "edc97239-052f-43c6-8595-858206808cf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:48:40.738113 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.738096 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "edc97239-052f-43c6-8595-858206808cf3" (UID: "edc97239-052f-43c6-8595-858206808cf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:48:40.738485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.738168 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "edc97239-052f-43c6-8595-858206808cf3" (UID: "edc97239-052f-43c6-8595-858206808cf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:48:40.738485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.738308 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "edc97239-052f-43c6-8595-858206808cf3" (UID: "edc97239-052f-43c6-8595-858206808cf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:48:40.738485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.738415 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-istio-data\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:48:40.738485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.738436 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-workload-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:48:40.738485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.738445 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc97239-052f-43c6-8595-858206808cf3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "edc97239-052f-43c6-8595-858206808cf3" (UID: "edc97239-052f-43c6-8595-858206808cf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:48:40.738485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.738465 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-credential-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:48:40.738485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.738485 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-workload-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:48:40.740027 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.740000 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc97239-052f-43c6-8595-858206808cf3-istio-token" (OuterVolumeSpecName: "istio-token") pod "edc97239-052f-43c6-8595-858206808cf3" (UID: "edc97239-052f-43c6-8595-858206808cf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:48:40.740152 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.740045 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "edc97239-052f-43c6-8595-858206808cf3" (UID: "edc97239-052f-43c6-8595-858206808cf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:48:40.740237 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.740221 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc97239-052f-43c6-8595-858206808cf3-kube-api-access-f25dq" (OuterVolumeSpecName: "kube-api-access-f25dq") pod "edc97239-052f-43c6-8595-858206808cf3" (UID: "edc97239-052f-43c6-8595-858206808cf3"). InnerVolumeSpecName "kube-api-access-f25dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:48:40.740301 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.740254 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edc97239-052f-43c6-8595-858206808cf3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "edc97239-052f-43c6-8595-858206808cf3" (UID: "edc97239-052f-43c6-8595-858206808cf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 15:48:40.839202 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.839171 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc97239-052f-43c6-8595-858206808cf3-istio-podinfo\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:48:40.839202 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.839199 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc97239-052f-43c6-8595-858206808cf3-istiod-ca-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:48:40.839202 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.839210 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc97239-052f-43c6-8595-858206808cf3-istio-envoy\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:48:40.839474 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.839218 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc97239-052f-43c6-8595-858206808cf3-istio-token\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:48:40.839474 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.839226 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f25dq\" (UniqueName: \"kubernetes.io/projected/edc97239-052f-43c6-8595-858206808cf3-kube-api-access-f25dq\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:48:40.906607 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:40.906576 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:41.314976 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:41.314903 2581 generic.go:356] "Generic (PLEG): container finished" podID="edc97239-052f-43c6-8595-858206808cf3" containerID="be0de193e5d5056395de0ade88265dcda9e91af111413d33b6d613b8255edf2c" exitCode=0 Sep 17 15:48:41.315151 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:41.314983 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" Sep 17 15:48:41.315151 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:41.315012 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" event={"ID":"edc97239-052f-43c6-8595-858206808cf3","Type":"ContainerDied","Data":"be0de193e5d5056395de0ade88265dcda9e91af111413d33b6d613b8255edf2c"} Sep 17 15:48:41.315151 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:41.315073 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6" event={"ID":"edc97239-052f-43c6-8595-858206808cf3","Type":"ContainerDied","Data":"3731de8d1b507a35f7a0880c52fbb45e857c7430c913504f2e247d1e9c50dd93"} Sep 17 15:48:41.315151 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:41.315096 2581 scope.go:117] "RemoveContainer" containerID="be0de193e5d5056395de0ade88265dcda9e91af111413d33b6d613b8255edf2c" Sep 17 15:48:41.324982 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:41.324963 2581 scope.go:117] "RemoveContainer" containerID="c99ef676aa921516e4aa3190cc5128703a64c192bc28a653d5e621e8656257d4" Sep 17 15:48:41.333232 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:41.333216 2581 scope.go:117] "RemoveContainer" containerID="be0de193e5d5056395de0ade88265dcda9e91af111413d33b6d613b8255edf2c" Sep 17 15:48:41.333488 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:48:41.333469 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0de193e5d5056395de0ade88265dcda9e91af111413d33b6d613b8255edf2c\": container with ID starting with be0de193e5d5056395de0ade88265dcda9e91af111413d33b6d613b8255edf2c not found: ID does not exist" containerID="be0de193e5d5056395de0ade88265dcda9e91af111413d33b6d613b8255edf2c" Sep 17 15:48:41.333563 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:41.333498 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0de193e5d5056395de0ade88265dcda9e91af111413d33b6d613b8255edf2c"} err="failed to get container status \"be0de193e5d5056395de0ade88265dcda9e91af111413d33b6d613b8255edf2c\": rpc error: code = NotFound desc = could not find container \"be0de193e5d5056395de0ade88265dcda9e91af111413d33b6d613b8255edf2c\": container with ID starting with be0de193e5d5056395de0ade88265dcda9e91af111413d33b6d613b8255edf2c not found: ID does not exist" Sep 17 15:48:41.333563 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:41.333520 2581 scope.go:117] "RemoveContainer" containerID="c99ef676aa921516e4aa3190cc5128703a64c192bc28a653d5e621e8656257d4" Sep 17 15:48:41.333763 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:48:41.333744 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99ef676aa921516e4aa3190cc5128703a64c192bc28a653d5e621e8656257d4\": container with ID starting with c99ef676aa921516e4aa3190cc5128703a64c192bc28a653d5e621e8656257d4 not found: ID does not exist" containerID="c99ef676aa921516e4aa3190cc5128703a64c192bc28a653d5e621e8656257d4" Sep 17 15:48:41.333804 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:41.333770 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99ef676aa921516e4aa3190cc5128703a64c192bc28a653d5e621e8656257d4"} err="failed to get container status \"c99ef676aa921516e4aa3190cc5128703a64c192bc28a653d5e621e8656257d4\": rpc error: code = NotFound desc = could not find container \"c99ef676aa921516e4aa3190cc5128703a64c192bc28a653d5e621e8656257d4\": container with ID starting with c99ef676aa921516e4aa3190cc5128703a64c192bc28a653d5e621e8656257d4 not found: ID does not exist" Sep 17 15:48:41.348312 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:41.348279 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6"] Sep 17 15:48:41.353241 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:41.353201 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-70207142-openshift-default-7b9c79db96-n4tj6"] Sep 17 15:48:41.464522 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:41.464483 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc97239-052f-43c6-8595-858206808cf3" path="/var/lib/kubelet/pods/edc97239-052f-43c6-8595-858206808cf3/volumes" Sep 17 15:48:43.285887 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:43.285853 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:43.600186 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:43.600106 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:43.699727 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:43.699697 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:43.832528 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:43.832496 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:44.066137 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:44.066102 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:44.485071 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:44.484993 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:45.621301 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:45.621267 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:46.408971 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:46.408881 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:47.416202 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:47.416168 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:48.263622 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:48.263587 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:49.683033 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.683003 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh"] Sep 17 15:48:49.683549 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.683529 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc97239-052f-43c6-8595-858206808cf3" containerName="istio-proxy" Sep 17 15:48:49.683632 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.683552 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc97239-052f-43c6-8595-858206808cf3" containerName="istio-proxy" Sep 17 15:48:49.683695 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.683677 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc97239-052f-43c6-8595-858206808cf3" containerName="istio-proxy" Sep 17 15:48:49.683695 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.683692 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc97239-052f-43c6-8595-858206808cf3" containerName="istio-proxy" Sep 17 15:48:49.685861 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.685840 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.689818 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.689799 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"distinct-w0-849b71-kserve-self-signed-certs\"" Sep 17 15:48:49.706085 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.706059 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh"] Sep 17 15:48:49.734861 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.734835 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:49.821467 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.821418 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-home\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.821467 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.821468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-dshm\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.821648 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.821528 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-tmp-dir\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.821648 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.821550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-model-cache\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.821648 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.821603 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25424ac6-a827-4f63-abb3-58a873ff27fa-tls-certs\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.821648 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.821641 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjt27\" (UniqueName: \"kubernetes.io/projected/25424ac6-a827-4f63-abb3-58a873ff27fa-kube-api-access-xjt27\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.922877 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.922847 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25424ac6-a827-4f63-abb3-58a873ff27fa-tls-certs\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.923035 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.922885 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjt27\" (UniqueName: \"kubernetes.io/projected/25424ac6-a827-4f63-abb3-58a873ff27fa-kube-api-access-xjt27\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.923090 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.923072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-home\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.923138 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.923105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-dshm\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.923207 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.923191 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-tmp-dir\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.923249 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.923226 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-model-cache\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.923527 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.923506 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-home\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.923629 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.923538 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-model-cache\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.923629 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.923593 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-tmp-dir\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.925134 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.925115 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-dshm\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.925221 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.925202 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25424ac6-a827-4f63-abb3-58a873ff27fa-tls-certs\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.933107 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.933051 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjt27\" (UniqueName: \"kubernetes.io/projected/25424ac6-a827-4f63-abb3-58a873ff27fa-kube-api-access-xjt27\") pod \"distinct-w0-849b71-kserve-76d76747fb-8hbwh\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:49.995027 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:49.995000 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:50.270592 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:50.270563 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh"] Sep 17 15:48:50.273035 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:48:50.273008 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25424ac6_a827_4f63_abb3_58a873ff27fa.slice/crio-fae10e79e38d17feea26a6938c9f032a3a31dd4cf870cb5088f22cd628885b08 WatchSource:0}: Error finding container fae10e79e38d17feea26a6938c9f032a3a31dd4cf870cb5088f22cd628885b08: Status 404 returned error can't find the container with id fae10e79e38d17feea26a6938c9f032a3a31dd4cf870cb5088f22cd628885b08 Sep 17 15:48:50.366894 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:50.366858 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" event={"ID":"25424ac6-a827-4f63-abb3-58a873ff27fa","Type":"ContainerStarted","Data":"c25a8e17d3a8195e864baab455c283f181b5ebbdb5d574dd1df9531467f18773"} Sep 17 15:48:50.367065 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:50.366899 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" event={"ID":"25424ac6-a827-4f63-abb3-58a873ff27fa","Type":"ContainerStarted","Data":"fae10e79e38d17feea26a6938c9f032a3a31dd4cf870cb5088f22cd628885b08"} Sep 17 15:48:50.390388 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:50.390341 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" podStartSLOduration=1.390324738 podStartE2EDuration="1.390324738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:50.390074112 +0000 UTC m=+1117.386310859" watchObservedRunningTime="2026-09-17 15:48:50.390324738 +0000 UTC m=+1117.386561481" Sep 17 15:48:51.166125 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:51.166096 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:51.459790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:51.459640 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:52.215206 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:52.215173 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:52.407135 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:52.407105 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:52.796590 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:52.796556 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:52.937238 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:52.937196 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:53.556236 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:53.556207 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:53.683368 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:53.683308 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:53.719351 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:53.719318 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp"] Sep 17 15:48:53.719603 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:53.719561 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" podUID="4ba57349-9d6c-471e-988d-7083ad61d916" containerName="payload-processing" containerID="cri-o://53e4b7afc913211ae4b2210f17fa635deb1d26f8169dff2bf6dd1259440c4192" gracePeriod=30 Sep 17 15:48:53.730862 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:53.730835 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr"] Sep 17 15:48:53.731088 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:53.731062 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" podUID="8b7e10e0-a030-47f6-9a61-14d9e30093e9" containerName="payload-pre-processing" containerID="cri-o://024909da84d8514877fdaf74cad5d8ffa1d26e56fd57bc8b751a618f93e69c0f" gracePeriod=30 Sep 17 15:48:53.773961 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:53.773936 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:53.897497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:53.897407 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:54.011040 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.011017 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" Sep 17 15:48:54.014023 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.014004 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" Sep 17 15:48:54.038571 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.038542 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:54.161957 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.161858 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:54.164075 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.164042 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z458\" (UniqueName: \"kubernetes.io/projected/4ba57349-9d6c-471e-988d-7083ad61d916-kube-api-access-8z458\") pod \"4ba57349-9d6c-471e-988d-7083ad61d916\" (UID: \"4ba57349-9d6c-471e-988d-7083ad61d916\") " Sep 17 15:48:54.164194 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.164137 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b7e10e0-a030-47f6-9a61-14d9e30093e9-plugins-config-volume\") pod \"8b7e10e0-a030-47f6-9a61-14d9e30093e9\" (UID: \"8b7e10e0-a030-47f6-9a61-14d9e30093e9\") " Sep 17 15:48:54.164249 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.164235 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba57349-9d6c-471e-988d-7083ad61d916-plugins-config-volume\") pod \"4ba57349-9d6c-471e-988d-7083ad61d916\" (UID: \"4ba57349-9d6c-471e-988d-7083ad61d916\") " Sep 17 15:48:54.164301 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.164287 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fspnp\" (UniqueName: \"kubernetes.io/projected/8b7e10e0-a030-47f6-9a61-14d9e30093e9-kube-api-access-fspnp\") pod \"8b7e10e0-a030-47f6-9a61-14d9e30093e9\" (UID: \"8b7e10e0-a030-47f6-9a61-14d9e30093e9\") " Sep 17 15:48:54.164608 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.164579 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7e10e0-a030-47f6-9a61-14d9e30093e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b7e10e0-a030-47f6-9a61-14d9e30093e9" (UID: "8b7e10e0-a030-47f6-9a61-14d9e30093e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:48:54.164608 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.164582 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba57349-9d6c-471e-988d-7083ad61d916-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ba57349-9d6c-471e-988d-7083ad61d916" (UID: "4ba57349-9d6c-471e-988d-7083ad61d916"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:48:54.166249 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.166222 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba57349-9d6c-471e-988d-7083ad61d916-kube-api-access-8z458" (OuterVolumeSpecName: "kube-api-access-8z458") pod "4ba57349-9d6c-471e-988d-7083ad61d916" (UID: "4ba57349-9d6c-471e-988d-7083ad61d916"). InnerVolumeSpecName "kube-api-access-8z458". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:48:54.166332 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.166296 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7e10e0-a030-47f6-9a61-14d9e30093e9-kube-api-access-fspnp" (OuterVolumeSpecName: "kube-api-access-fspnp") pod "8b7e10e0-a030-47f6-9a61-14d9e30093e9" (UID: "8b7e10e0-a030-47f6-9a61-14d9e30093e9"). InnerVolumeSpecName "kube-api-access-fspnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:48:54.265239 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.265198 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba57349-9d6c-471e-988d-7083ad61d916-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:48:54.265239 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.265231 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fspnp\" (UniqueName: \"kubernetes.io/projected/8b7e10e0-a030-47f6-9a61-14d9e30093e9-kube-api-access-fspnp\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:48:54.265239 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.265244 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z458\" (UniqueName: \"kubernetes.io/projected/4ba57349-9d6c-471e-988d-7083ad61d916-kube-api-access-8z458\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:48:54.265502 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.265256 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b7e10e0-a030-47f6-9a61-14d9e30093e9-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:48:54.308768 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.308739 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:54.391076 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.391044 2581 generic.go:356] "Generic (PLEG): container finished" podID="4ba57349-9d6c-471e-988d-7083ad61d916" containerID="53e4b7afc913211ae4b2210f17fa635deb1d26f8169dff2bf6dd1259440c4192" exitCode=0 Sep 17 15:48:54.391262 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.391090 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" Sep 17 15:48:54.391262 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.391133 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" event={"ID":"4ba57349-9d6c-471e-988d-7083ad61d916","Type":"ContainerDied","Data":"53e4b7afc913211ae4b2210f17fa635deb1d26f8169dff2bf6dd1259440c4192"} Sep 17 15:48:54.391262 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.391196 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp" event={"ID":"4ba57349-9d6c-471e-988d-7083ad61d916","Type":"ContainerDied","Data":"d90deeadad4f187eaa752a410c6df4103cdae0466e74009c5a7855d0d17ca71d"} Sep 17 15:48:54.391262 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.391229 2581 scope.go:117] "RemoveContainer" containerID="53e4b7afc913211ae4b2210f17fa635deb1d26f8169dff2bf6dd1259440c4192" Sep 17 15:48:54.392627 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.392599 2581 generic.go:356] "Generic (PLEG): container finished" podID="8b7e10e0-a030-47f6-9a61-14d9e30093e9" containerID="024909da84d8514877fdaf74cad5d8ffa1d26e56fd57bc8b751a618f93e69c0f" exitCode=0 Sep 17 15:48:54.392733 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.392647 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" Sep 17 15:48:54.392795 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.392724 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" event={"ID":"8b7e10e0-a030-47f6-9a61-14d9e30093e9","Type":"ContainerDied","Data":"024909da84d8514877fdaf74cad5d8ffa1d26e56fd57bc8b751a618f93e69c0f"} Sep 17 15:48:54.392795 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.392777 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr" event={"ID":"8b7e10e0-a030-47f6-9a61-14d9e30093e9","Type":"ContainerDied","Data":"f6a331c6bb89e2476a6f420d501e6493b94b9fdc6c7da7becebac7b4b509eb0c"} Sep 17 15:48:54.402261 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.402242 2581 scope.go:117] "RemoveContainer" containerID="53e4b7afc913211ae4b2210f17fa635deb1d26f8169dff2bf6dd1259440c4192" Sep 17 15:48:54.402702 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:48:54.402679 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e4b7afc913211ae4b2210f17fa635deb1d26f8169dff2bf6dd1259440c4192\": container with ID starting with 53e4b7afc913211ae4b2210f17fa635deb1d26f8169dff2bf6dd1259440c4192 not found: ID does not exist" containerID="53e4b7afc913211ae4b2210f17fa635deb1d26f8169dff2bf6dd1259440c4192" Sep 17 15:48:54.402792 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.402712 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e4b7afc913211ae4b2210f17fa635deb1d26f8169dff2bf6dd1259440c4192"} err="failed to get container status \"53e4b7afc913211ae4b2210f17fa635deb1d26f8169dff2bf6dd1259440c4192\": rpc error: code = NotFound desc = could not find container \"53e4b7afc913211ae4b2210f17fa635deb1d26f8169dff2bf6dd1259440c4192\": container with ID starting with 53e4b7afc913211ae4b2210f17fa635deb1d26f8169dff2bf6dd1259440c4192 not found: ID does not exist" Sep 17 15:48:54.402792 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.402735 2581 scope.go:117] "RemoveContainer" containerID="024909da84d8514877fdaf74cad5d8ffa1d26e56fd57bc8b751a618f93e69c0f" Sep 17 15:48:54.410456 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.410438 2581 scope.go:117] "RemoveContainer" containerID="024909da84d8514877fdaf74cad5d8ffa1d26e56fd57bc8b751a618f93e69c0f" Sep 17 15:48:54.410691 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:48:54.410676 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024909da84d8514877fdaf74cad5d8ffa1d26e56fd57bc8b751a618f93e69c0f\": container with ID starting with 024909da84d8514877fdaf74cad5d8ffa1d26e56fd57bc8b751a618f93e69c0f not found: ID does not exist" containerID="024909da84d8514877fdaf74cad5d8ffa1d26e56fd57bc8b751a618f93e69c0f" Sep 17 15:48:54.410745 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.410695 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024909da84d8514877fdaf74cad5d8ffa1d26e56fd57bc8b751a618f93e69c0f"} err="failed to get container status \"024909da84d8514877fdaf74cad5d8ffa1d26e56fd57bc8b751a618f93e69c0f\": rpc error: code = NotFound desc = could not find container \"024909da84d8514877fdaf74cad5d8ffa1d26e56fd57bc8b751a618f93e69c0f\": container with ID starting with 024909da84d8514877fdaf74cad5d8ffa1d26e56fd57bc8b751a618f93e69c0f not found: ID does not exist" Sep 17 15:48:54.428433 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.428372 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr"] Sep 17 15:48:54.431335 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.431314 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-449c47-56b675594b-42djr"] Sep 17 15:48:54.459271 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.459218 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp"] Sep 17 15:48:54.463587 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.463565 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:54.469395 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.469375 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-449c47-85fcd6c879-8r6wp"] Sep 17 15:48:54.700192 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:54.700115 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:55.461956 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:55.461896 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba57349-9d6c-471e-988d-7083ad61d916" path="/var/lib/kubelet/pods/4ba57349-9d6c-471e-988d-7083ad61d916/volumes" Sep 17 15:48:55.462320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:55.462306 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b7e10e0-a030-47f6-9a61-14d9e30093e9" path="/var/lib/kubelet/pods/8b7e10e0-a030-47f6-9a61-14d9e30093e9/volumes" Sep 17 15:48:55.648858 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:55.648824 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:55.912950 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:55.912894 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:56.011445 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:56.011414 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:56.585309 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:56.585277 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:56.740978 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:56.740942 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:56.811751 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:56.811702 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df"] Sep 17 15:48:56.812002 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:56.811974 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" podUID="b8557711-d68e-4763-b8db-abef7b4d0452" containerName="istio-proxy" containerID="cri-o://a7419c92d5612e69446eae046ecb642c9231337d1e90f46af879071a0d741872" gracePeriod=30 Sep 17 15:48:56.870982 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:56.870883 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:57.041338 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:57.041299 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:57.155094 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:57.155005 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:57.257006 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:57.256972 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:57.491767 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:57.491686 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:58.263715 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:58.263684 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:58.363167 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:58.363135 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:58.471189 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:58.471156 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:58.886127 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:58.886094 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:58.988631 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:58.988598 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:59.078408 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.078377 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr"] Sep 17 15:48:59.078738 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.078726 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba57349-9d6c-471e-988d-7083ad61d916" containerName="payload-processing" Sep 17 15:48:59.078790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.078739 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba57349-9d6c-471e-988d-7083ad61d916" containerName="payload-processing" Sep 17 15:48:59.078790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.078757 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc97239-052f-43c6-8595-858206808cf3" containerName="istio-proxy" Sep 17 15:48:59.078790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.078763 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc97239-052f-43c6-8595-858206808cf3" containerName="istio-proxy" Sep 17 15:48:59.078790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.078776 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7e10e0-a030-47f6-9a61-14d9e30093e9" containerName="payload-pre-processing" Sep 17 15:48:59.078790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.078782 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7e10e0-a030-47f6-9a61-14d9e30093e9" containerName="payload-pre-processing" Sep 17 15:48:59.078963 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.078838 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba57349-9d6c-471e-988d-7083ad61d916" containerName="payload-processing" Sep 17 15:48:59.078963 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.078847 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7e10e0-a030-47f6-9a61-14d9e30093e9" containerName="payload-pre-processing" Sep 17 15:48:59.081743 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.081728 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" Sep 17 15:48:59.084031 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.084008 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2b23c7\"" Sep 17 15:48:59.084614 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.084600 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2b23c7-dockercfg-q6nmr\"" Sep 17 15:48:59.093526 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.093502 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr"] Sep 17 15:48:59.209740 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.209633 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff6d1d6-ca57-43d7-a213-4451821f437a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr\" (UID: \"6ff6d1d6-ca57-43d7-a213-4451821f437a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" Sep 17 15:48:59.209740 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.209703 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg8sw\" (UniqueName: \"kubernetes.io/projected/6ff6d1d6-ca57-43d7-a213-4451821f437a-kube-api-access-vg8sw\") pod \"payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr\" (UID: \"6ff6d1d6-ca57-43d7-a213-4451821f437a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" Sep 17 15:48:59.310973 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.310937 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff6d1d6-ca57-43d7-a213-4451821f437a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr\" (UID: \"6ff6d1d6-ca57-43d7-a213-4451821f437a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" Sep 17 15:48:59.311384 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.311011 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg8sw\" (UniqueName: \"kubernetes.io/projected/6ff6d1d6-ca57-43d7-a213-4451821f437a-kube-api-access-vg8sw\") pod \"payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr\" (UID: \"6ff6d1d6-ca57-43d7-a213-4451821f437a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" Sep 17 15:48:59.311683 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.311661 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff6d1d6-ca57-43d7-a213-4451821f437a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr\" (UID: \"6ff6d1d6-ca57-43d7-a213-4451821f437a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" Sep 17 15:48:59.322386 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.322361 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg8sw\" (UniqueName: \"kubernetes.io/projected/6ff6d1d6-ca57-43d7-a213-4451821f437a-kube-api-access-vg8sw\") pod \"payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr\" (UID: \"6ff6d1d6-ca57-43d7-a213-4451821f437a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" Sep 17 15:48:59.393218 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.393181 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" Sep 17 15:48:59.511590 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.511519 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:59.675255 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.675225 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:48:59.685085 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.685060 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr"] Sep 17 15:48:59.687718 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:48:59.687686 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff6d1d6_ca57_43d7_a213_4451821f437a.slice/crio-71d5df7ce98958db639650b68268a1fb436df815539e99e39edf2b2f89d02fbc WatchSource:0}: Error finding container 71d5df7ce98958db639650b68268a1fb436df815539e99e39edf2b2f89d02fbc: Status 404 returned error can't find the container with id 71d5df7ce98958db639650b68268a1fb436df815539e99e39edf2b2f89d02fbc Sep 17 15:48:59.995553 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.995508 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:48:59.995755 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:48:59.995587 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:49:00.008256 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:00.008230 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:49:00.432332 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:00.432294 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" event={"ID":"6ff6d1d6-ca57-43d7-a213-4451821f437a","Type":"ContainerStarted","Data":"a60c6f82f751532403ddd18521070d6161354850a95fab95ecdbf1d1e5c2674a"} Sep 17 15:49:00.432810 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:00.432362 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" event={"ID":"6ff6d1d6-ca57-43d7-a213-4451821f437a","Type":"ContainerStarted","Data":"71d5df7ce98958db639650b68268a1fb436df815539e99e39edf2b2f89d02fbc"} Sep 17 15:49:00.432810 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:00.432469 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" Sep 17 15:49:00.445904 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:00.445874 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:49:00.481363 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:00.481304 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" podStartSLOduration=1.48128237 podStartE2EDuration="1.48128237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:00.479849303 +0000 UTC m=+1127.476086050" watchObservedRunningTime="2026-09-17 15:49:00.48128237 +0000 UTC m=+1127.477519119" Sep 17 15:49:00.552979 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:00.552950 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:02.063150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.063121 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:49:02.139841 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.139738 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-credential-socket\") pod \"b8557711-d68e-4763-b8db-abef7b4d0452\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " Sep 17 15:49:02.139841 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.139812 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8557711-d68e-4763-b8db-abef7b4d0452-istiod-ca-cert\") pod \"b8557711-d68e-4763-b8db-abef7b4d0452\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " Sep 17 15:49:02.140291 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.139891 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8557711-d68e-4763-b8db-abef7b4d0452-istio-podinfo\") pod \"b8557711-d68e-4763-b8db-abef7b4d0452\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " Sep 17 15:49:02.140291 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.140083 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-istio-envoy\") pod \"b8557711-d68e-4763-b8db-abef7b4d0452\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " Sep 17 15:49:02.140291 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.140155 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8557711-d68e-4763-b8db-abef7b4d0452-istio-token\") pod \"b8557711-d68e-4763-b8db-abef7b4d0452\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " Sep 17 15:49:02.140291 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.140178 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b8557711-d68e-4763-b8db-abef7b4d0452" (UID: "b8557711-d68e-4763-b8db-abef7b4d0452"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:49:02.140291 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.140191 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-workload-socket\") pod \"b8557711-d68e-4763-b8db-abef7b4d0452\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " Sep 17 15:49:02.140291 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.140271 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-istio-data\") pod \"b8557711-d68e-4763-b8db-abef7b4d0452\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " Sep 17 15:49:02.140677 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.140298 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-workload-certs\") pod \"b8557711-d68e-4763-b8db-abef7b4d0452\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " Sep 17 15:49:02.140677 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.140365 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx586\" (UniqueName: \"kubernetes.io/projected/b8557711-d68e-4763-b8db-abef7b4d0452-kube-api-access-sx586\") pod \"b8557711-d68e-4763-b8db-abef7b4d0452\" (UID: \"b8557711-d68e-4763-b8db-abef7b4d0452\") " Sep 17 15:49:02.140677 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.140512 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8557711-d68e-4763-b8db-abef7b4d0452-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b8557711-d68e-4763-b8db-abef7b4d0452" (UID: "b8557711-d68e-4763-b8db-abef7b4d0452"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:49:02.140827 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.140731 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b8557711-d68e-4763-b8db-abef7b4d0452" (UID: "b8557711-d68e-4763-b8db-abef7b4d0452"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:49:02.140827 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.140758 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-credential-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:02.140827 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.140779 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8557711-d68e-4763-b8db-abef7b4d0452-istiod-ca-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:02.140827 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.140801 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-istio-data" (OuterVolumeSpecName: "istio-data") pod "b8557711-d68e-4763-b8db-abef7b4d0452" (UID: "b8557711-d68e-4763-b8db-abef7b4d0452"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:49:02.141077 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.140870 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b8557711-d68e-4763-b8db-abef7b4d0452" (UID: "b8557711-d68e-4763-b8db-abef7b4d0452"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:49:02.142480 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.142456 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b8557711-d68e-4763-b8db-abef7b4d0452" (UID: "b8557711-d68e-4763-b8db-abef7b4d0452"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:49:02.142480 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.142461 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8557711-d68e-4763-b8db-abef7b4d0452-istio-token" (OuterVolumeSpecName: "istio-token") pod "b8557711-d68e-4763-b8db-abef7b4d0452" (UID: "b8557711-d68e-4763-b8db-abef7b4d0452"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:49:02.142668 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.142651 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8557711-d68e-4763-b8db-abef7b4d0452-kube-api-access-sx586" (OuterVolumeSpecName: "kube-api-access-sx586") pod "b8557711-d68e-4763-b8db-abef7b4d0452" (UID: "b8557711-d68e-4763-b8db-abef7b4d0452"). InnerVolumeSpecName "kube-api-access-sx586". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:49:02.142830 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.142817 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8557711-d68e-4763-b8db-abef7b4d0452-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b8557711-d68e-4763-b8db-abef7b4d0452" (UID: "b8557711-d68e-4763-b8db-abef7b4d0452"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 15:49:02.187879 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.187843 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:02.242055 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.242024 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8557711-d68e-4763-b8db-abef7b4d0452-istio-podinfo\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:02.242055 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.242049 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-istio-envoy\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:02.242055 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.242057 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8557711-d68e-4763-b8db-abef7b4d0452-istio-token\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:02.242281 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.242066 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-workload-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:02.242281 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.242074 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-istio-data\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:02.242281 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.242085 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8557711-d68e-4763-b8db-abef7b4d0452-workload-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:02.242281 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.242093 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx586\" (UniqueName: \"kubernetes.io/projected/b8557711-d68e-4763-b8db-abef7b4d0452-kube-api-access-sx586\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:02.444819 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.444734 2581 generic.go:356] "Generic (PLEG): container finished" podID="b8557711-d68e-4763-b8db-abef7b4d0452" containerID="a7419c92d5612e69446eae046ecb642c9231337d1e90f46af879071a0d741872" exitCode=0 Sep 17 15:49:02.444819 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.444779 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" Sep 17 15:49:02.445021 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.444819 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" event={"ID":"b8557711-d68e-4763-b8db-abef7b4d0452","Type":"ContainerDied","Data":"a7419c92d5612e69446eae046ecb642c9231337d1e90f46af879071a0d741872"} Sep 17 15:49:02.445021 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.444870 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df" event={"ID":"b8557711-d68e-4763-b8db-abef7b4d0452","Type":"ContainerDied","Data":"45777f80f72664b24d6136de49ed60b55f881de2694667b0e9f64550753cd8b0"} Sep 17 15:49:02.445021 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.444892 2581 scope.go:117] "RemoveContainer" containerID="a7419c92d5612e69446eae046ecb642c9231337d1e90f46af879071a0d741872" Sep 17 15:49:02.454619 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.454601 2581 scope.go:117] "RemoveContainer" containerID="a7419c92d5612e69446eae046ecb642c9231337d1e90f46af879071a0d741872" Sep 17 15:49:02.454955 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:02.454906 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7419c92d5612e69446eae046ecb642c9231337d1e90f46af879071a0d741872\": container with ID starting with a7419c92d5612e69446eae046ecb642c9231337d1e90f46af879071a0d741872 not found: ID does not exist" containerID="a7419c92d5612e69446eae046ecb642c9231337d1e90f46af879071a0d741872" Sep 17 15:49:02.455049 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.454964 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7419c92d5612e69446eae046ecb642c9231337d1e90f46af879071a0d741872"} err="failed to get container status \"a7419c92d5612e69446eae046ecb642c9231337d1e90f46af879071a0d741872\": rpc error: code = NotFound desc = could not find container \"a7419c92d5612e69446eae046ecb642c9231337d1e90f46af879071a0d741872\": container with ID starting with a7419c92d5612e69446eae046ecb642c9231337d1e90f46af879071a0d741872 not found: ID does not exist" Sep 17 15:49:02.482625 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.482591 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df"] Sep 17 15:49:02.495290 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:02.495262 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-d9afa3-openshift-default-f8849555b-zg5df"] Sep 17 15:49:03.462540 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:03.462509 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8557711-d68e-4763-b8db-abef7b4d0452" path="/var/lib/kubelet/pods/b8557711-d68e-4763-b8db-abef7b4d0452/volumes" Sep 17 15:49:10.689681 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:10.689645 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:11.440826 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:11.440794 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" Sep 17 15:49:13.701665 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:13.701622 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:19.355111 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:19.355062 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:20.680637 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:20.680601 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:20.781188 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:20.781154 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:21.155682 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.155652 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:21.473205 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.473131 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq"] Sep 17 15:49:21.473486 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.473473 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8557711-d68e-4763-b8db-abef7b4d0452" containerName="istio-proxy" Sep 17 15:49:21.473529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.473488 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8557711-d68e-4763-b8db-abef7b4d0452" containerName="istio-proxy" Sep 17 15:49:21.473569 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.473560 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8557711-d68e-4763-b8db-abef7b4d0452" containerName="istio-proxy" Sep 17 15:49:21.478025 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.478009 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.480761 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.480737 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-608b97-openshift-default-dockercfg-rjhkm\"" Sep 17 15:49:21.504957 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.504927 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq"] Sep 17 15:49:21.511067 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.511038 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-envoy\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.511216 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.511071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-podinfo\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.511216 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.511145 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-data\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.511216 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.511185 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ae7f28-a274-491b-8c4d-6c88612e9601-istiod-ca-cert\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.511407 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.511245 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-credential-socket\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.511407 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.511344 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-workload-socket\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.511407 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.511367 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrjvt\" (UniqueName: \"kubernetes.io/projected/f9ae7f28-a274-491b-8c4d-6c88612e9601-kube-api-access-wrjvt\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.511581 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.511423 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-workload-certs\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.511581 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.511438 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-token\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.531410 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.531382 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:21.612527 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.612494 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ae7f28-a274-491b-8c4d-6c88612e9601-istiod-ca-cert\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.612732 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.612540 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-credential-socket\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.612732 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.612588 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-workload-socket\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.612732 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.612612 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrjvt\" (UniqueName: \"kubernetes.io/projected/f9ae7f28-a274-491b-8c4d-6c88612e9601-kube-api-access-wrjvt\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.612732 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.612674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-workload-certs\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.612732 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.612696 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-token\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.613058 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.612781 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-envoy\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.613058 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.612820 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-podinfo\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.613058 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.612854 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-data\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.613058 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.613013 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-credential-socket\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.613274 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.613081 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-workload-socket\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.613274 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.613238 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-workload-certs\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.613371 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.613279 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-data\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.613371 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.613359 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ae7f28-a274-491b-8c4d-6c88612e9601-istiod-ca-cert\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.615093 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.615072 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-envoy\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.615333 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.615316 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-podinfo\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.622378 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.622354 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrjvt\" (UniqueName: \"kubernetes.io/projected/f9ae7f28-a274-491b-8c4d-6c88612e9601-kube-api-access-wrjvt\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.627600 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.627582 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-token\") pod \"e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:21.714244 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.714215 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:21.788536 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:21.788512 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:22.086065 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:22.085938 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq"] Sep 17 15:49:22.088933 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:49:22.088895 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ae7f28_a274_491b_8c4d_6c88612e9601.slice/crio-f624eff8f1f090090d4ea60f9c9d10443962b668dd2d47e0a46a84859f315291 WatchSource:0}: Error finding container f624eff8f1f090090d4ea60f9c9d10443962b668dd2d47e0a46a84859f315291: Status 404 returned error can't find the container with id f624eff8f1f090090d4ea60f9c9d10443962b668dd2d47e0a46a84859f315291 Sep 17 15:49:22.091212 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:22.091179 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:49:22.091315 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:22.091245 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:49:22.091315 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:22.091274 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:49:22.187315 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:22.187270 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:22.556672 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:22.556631 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" event={"ID":"f9ae7f28-a274-491b-8c4d-6c88612e9601","Type":"ContainerStarted","Data":"7df4b262a8a943d2fcb6bcad2c27dc369033c949a1c7b8a263928bd7be4f0706"} Sep 17 15:49:22.556865 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:22.556688 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" event={"ID":"f9ae7f28-a274-491b-8c4d-6c88612e9601","Type":"ContainerStarted","Data":"f624eff8f1f090090d4ea60f9c9d10443962b668dd2d47e0a46a84859f315291"} Sep 17 15:49:22.583225 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:22.583174 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" podStartSLOduration=1.583157084 podStartE2EDuration="1.583157084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:22.58070637 +0000 UTC m=+1149.576943128" watchObservedRunningTime="2026-09-17 15:49:22.583157084 +0000 UTC m=+1149.579393865" Sep 17 15:49:22.761548 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:22.761514 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:22.789455 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:22.789430 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:23.795216 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:23.795187 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:24.050217 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:24.050136 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:24.567441 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:24.567409 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:24.568225 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:24.568205 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:24.616707 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:24.616675 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:24.711029 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:24.710995 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:25.012669 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:25.012585 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:25.902659 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:25.902622 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:26.000813 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:26.000778 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:26.147967 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:26.147931 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:26.746706 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:26.746676 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:26.849091 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:26.849055 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:27.103140 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:27.103107 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:27.984443 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:27.984410 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:28.769725 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:28.769692 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:29.447191 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:29.447153 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:31.838978 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:31.838941 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:37.657158 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:37.657083 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:38.053174 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:38.053139 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:38.679386 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:38.679336 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:39.922441 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:39.922395 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:40.820104 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:40.820074 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:41.033169 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:41.033127 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:41.906813 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:41.906775 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:42.915778 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:42.915744 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:43.669814 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:43.669780 2581 generic.go:356] "Generic (PLEG): container finished" podID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerID="7df4b262a8a943d2fcb6bcad2c27dc369033c949a1c7b8a263928bd7be4f0706" exitCode=0 Sep 17 15:49:43.670034 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:43.669851 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" event={"ID":"f9ae7f28-a274-491b-8c4d-6c88612e9601","Type":"ContainerDied","Data":"7df4b262a8a943d2fcb6bcad2c27dc369033c949a1c7b8a263928bd7be4f0706"} Sep 17 15:49:43.670286 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:43.670272 2581 scope.go:117] "RemoveContainer" containerID="7df4b262a8a943d2fcb6bcad2c27dc369033c949a1c7b8a263928bd7be4f0706" Sep 17 15:49:43.671390 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:43.671363 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:49:43.671444 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:43.671428 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:49:43.671479 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:43.671456 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:49:43.691518 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:43.691496 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:44.473645 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:44.473596 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:44.582075 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:44.582043 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:44.677647 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:44.677614 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" event={"ID":"f9ae7f28-a274-491b-8c4d-6c88612e9601","Type":"ContainerStarted","Data":"ee27c59dd121559ebd9a5a122b06ddecba1342681b55705f304f6ee66f2b7fba"} Sep 17 15:49:44.788875 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:44.788844 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:44.789053 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:44.789008 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 17 15:49:44.789102 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:44.789057 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 17 15:49:45.277542 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:45.277502 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7788-9586b5"] Sep 17 15:49:45.281001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:45.280974 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-9586b5" Sep 17 15:49:45.289601 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:45.289574 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-55clv\"" Sep 17 15:49:45.289959 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:45.289930 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7788-9586b5"] Sep 17 15:49:45.397528 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:45.397493 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:45.435693 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:45.435661 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fbsf\" (UniqueName: \"kubernetes.io/projected/0a424c04-ef8c-49d7-ae01-2fc346f79013-kube-api-access-7fbsf\") pod \"test-curl-7788-9586b5\" (UID: \"0a424c04-ef8c-49d7-ae01-2fc346f79013\") " pod="opendatahub/test-curl-7788-9586b5" Sep 17 15:49:45.537244 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:45.537151 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fbsf\" (UniqueName: \"kubernetes.io/projected/0a424c04-ef8c-49d7-ae01-2fc346f79013-kube-api-access-7fbsf\") pod \"test-curl-7788-9586b5\" (UID: \"0a424c04-ef8c-49d7-ae01-2fc346f79013\") " pod="opendatahub/test-curl-7788-9586b5" Sep 17 15:49:45.549257 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:45.549226 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fbsf\" (UniqueName: \"kubernetes.io/projected/0a424c04-ef8c-49d7-ae01-2fc346f79013-kube-api-access-7fbsf\") pod \"test-curl-7788-9586b5\" (UID: \"0a424c04-ef8c-49d7-ae01-2fc346f79013\") " pod="opendatahub/test-curl-7788-9586b5" Sep 17 15:49:45.607017 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:45.606984 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-9586b5" Sep 17 15:49:45.794003 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:45.790690 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 17 15:49:45.794003 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:45.790774 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 17 15:49:45.879148 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:45.879111 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7788-9586b5"] Sep 17 15:49:45.893380 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:49:45.893345 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a424c04_ef8c_49d7_ae01_2fc346f79013.slice/crio-69d1603b0d169c20fbb8306b5ed8c1d7fa3099c3776eea9bc96f5c9e2d8a3d72 WatchSource:0}: Error finding container 69d1603b0d169c20fbb8306b5ed8c1d7fa3099c3776eea9bc96f5c9e2d8a3d72: Status 404 returned error can't find the container with id 69d1603b0d169c20fbb8306b5ed8c1d7fa3099c3776eea9bc96f5c9e2d8a3d72 Sep 17 15:49:46.083696 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:46.083571 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:46.710092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:46.710052 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7788-9586b5" event={"ID":"0a424c04-ef8c-49d7-ae01-2fc346f79013","Type":"ContainerStarted","Data":"75d7e2633d72f3f1ed1e804be179cc16629b996e3ba7eacb4c27a524220519fe"} Sep 17 15:49:46.710478 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:46.710110 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7788-9586b5" event={"ID":"0a424c04-ef8c-49d7-ae01-2fc346f79013","Type":"ContainerStarted","Data":"69d1603b0d169c20fbb8306b5ed8c1d7fa3099c3776eea9bc96f5c9e2d8a3d72"} Sep 17 15:49:46.726775 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:46.726715 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7788-9586b5" podStartSLOduration=1.7266981829999999 podStartE2EDuration="1.726698183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:46.724956694 +0000 UTC m=+1173.721193441" watchObservedRunningTime="2026-09-17 15:49:46.726698183 +0000 UTC m=+1173.722934982" Sep 17 15:49:46.781260 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:46.781171 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:46.789832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:46.789421 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 17 15:49:46.789832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:46.789479 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 17 15:49:47.370928 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.370874 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7788-9586b5"] Sep 17 15:49:47.375623 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.375586 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7788-9586b5"] Sep 17 15:49:47.535265 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.535227 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7788-d17a7c"] Sep 17 15:49:47.536099 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.536072 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a424c04-ef8c-49d7-ae01-2fc346f79013" containerName="test-curl-7788-9586b5" Sep 17 15:49:47.536265 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.536253 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a424c04-ef8c-49d7-ae01-2fc346f79013" containerName="test-curl-7788-9586b5" Sep 17 15:49:47.536543 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.536529 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a424c04-ef8c-49d7-ae01-2fc346f79013" containerName="test-curl-7788-9586b5" Sep 17 15:49:47.544197 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.544170 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-d17a7c" Sep 17 15:49:47.552650 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.552613 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7788-d17a7c"] Sep 17 15:49:47.660521 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.660479 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjlcs\" (UniqueName: \"kubernetes.io/projected/ebd7686b-b95f-4237-bd2d-38593650a586-kube-api-access-mjlcs\") pod \"test-curl-7788-d17a7c\" (UID: \"ebd7686b-b95f-4237-bd2d-38593650a586\") " pod="opendatahub/test-curl-7788-d17a7c" Sep 17 15:49:47.716133 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:47.716104 2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7788-9586b5" container="test-curl-7788-9586b5" Sep 17 15:49:47.761152 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.761096 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjlcs\" (UniqueName: \"kubernetes.io/projected/ebd7686b-b95f-4237-bd2d-38593650a586-kube-api-access-mjlcs\") pod \"test-curl-7788-d17a7c\" (UID: \"ebd7686b-b95f-4237-bd2d-38593650a586\") " pod="opendatahub/test-curl-7788-d17a7c" Sep 17 15:49:47.772529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.772492 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjlcs\" (UniqueName: \"kubernetes.io/projected/ebd7686b-b95f-4237-bd2d-38593650a586-kube-api-access-mjlcs\") pod \"test-curl-7788-d17a7c\" (UID: \"ebd7686b-b95f-4237-bd2d-38593650a586\") " pod="opendatahub/test-curl-7788-d17a7c" Sep 17 15:49:47.789760 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.789683 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 17 15:49:47.790040 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.790001 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 17 15:49:47.857872 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:47.857838 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-d17a7c" Sep 17 15:49:48.023102 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:48.023059 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:48.153217 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:48.153192 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7788-d17a7c"] Sep 17 15:49:48.154931 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:49:48.154892 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebd7686b_b95f_4237_bd2d_38593650a586.slice/crio-73210e0aecbd8e1f0766182301f12c57b73aeb956ba71fa899280dec1ab2c1d9 WatchSource:0}: Error finding container 73210e0aecbd8e1f0766182301f12c57b73aeb956ba71fa899280dec1ab2c1d9: Status 404 returned error can't find the container with id 73210e0aecbd8e1f0766182301f12c57b73aeb956ba71fa899280dec1ab2c1d9 Sep 17 15:49:48.493737 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:48.493638 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:48.723243 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:48.723202 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7788-d17a7c" event={"ID":"ebd7686b-b95f-4237-bd2d-38593650a586","Type":"ContainerStarted","Data":"1cce9162430f1720b084c97371eaa8f353dda56a196e48080a39ab8750339fec"} Sep 17 15:49:48.723666 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:48.723253 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7788-d17a7c" event={"ID":"ebd7686b-b95f-4237-bd2d-38593650a586","Type":"ContainerStarted","Data":"73210e0aecbd8e1f0766182301f12c57b73aeb956ba71fa899280dec1ab2c1d9"} Sep 17 15:49:48.723666 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:48.723329 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7788-9586b5" podUID="0a424c04-ef8c-49d7-ae01-2fc346f79013" containerName="test-curl-7788-9586b5" containerID="cri-o://75d7e2633d72f3f1ed1e804be179cc16629b996e3ba7eacb4c27a524220519fe" gracePeriod=2 Sep 17 15:49:48.740757 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:48.740707 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7788-d17a7c" podStartSLOduration=1.740691731 podStartE2EDuration="1.740691731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:48.739816708 +0000 UTC m=+1175.736053455" watchObservedRunningTime="2026-09-17 15:49:48.740691731 +0000 UTC m=+1175.736928523" Sep 17 15:49:48.790548 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:48.790510 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 17 15:49:48.790747 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:48.790583 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 17 15:49:49.212397 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.212361 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:49.417799 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.417713 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7788-d17a7c"] Sep 17 15:49:49.421206 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.421179 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7788-d17a7c"] Sep 17 15:49:49.546567 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.546528 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7788-77bf5a"] Sep 17 15:49:49.546948 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.546933 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd7686b-b95f-4237-bd2d-38593650a586" containerName="test-curl-7788-d17a7c" Sep 17 15:49:49.547000 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.546949 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd7686b-b95f-4237-bd2d-38593650a586" containerName="test-curl-7788-d17a7c" Sep 17 15:49:49.547037 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.547011 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd7686b-b95f-4237-bd2d-38593650a586" containerName="test-curl-7788-d17a7c" Sep 17 15:49:49.549813 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.549792 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-77bf5a" Sep 17 15:49:49.556035 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.556006 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7788-77bf5a"] Sep 17 15:49:49.696496 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.696383 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn26g\" (UniqueName: \"kubernetes.io/projected/cfdfed90-bc9b-4295-97b5-225e53185205-kube-api-access-bn26g\") pod \"test-curl-7788-77bf5a\" (UID: \"cfdfed90-bc9b-4295-97b5-225e53185205\") " pod="opendatahub/test-curl-7788-77bf5a" Sep 17 15:49:49.729371 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:49.729345 2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7788-d17a7c" container="test-curl-7788-d17a7c" Sep 17 15:49:49.790937 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.788233 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll"] Sep 17 15:49:49.790937 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.788477 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" podUID="57f6762f-ad32-458d-88b3-34ccb6232990" containerName="maas-api" containerID="cri-o://bc5082140c8433d3f0b6b2e8333f6d802dc07714f7ccb2499cb8d5c0f3897853" gracePeriod=30 Sep 17 15:49:49.790937 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.789038 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 17 15:49:49.790937 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.789105 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 17 15:49:49.805492 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.798105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn26g\" (UniqueName: \"kubernetes.io/projected/cfdfed90-bc9b-4295-97b5-225e53185205-kube-api-access-bn26g\") pod \"test-curl-7788-77bf5a\" (UID: \"cfdfed90-bc9b-4295-97b5-225e53185205\") " pod="opendatahub/test-curl-7788-77bf5a" Sep 17 15:49:49.805492 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.804185 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:49.817214 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.817158 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn26g\" (UniqueName: \"kubernetes.io/projected/cfdfed90-bc9b-4295-97b5-225e53185205-kube-api-access-bn26g\") pod \"test-curl-7788-77bf5a\" (UID: \"cfdfed90-bc9b-4295-97b5-225e53185205\") " pod="opendatahub/test-curl-7788-77bf5a" Sep 17 15:49:49.823312 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.823283 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k"] Sep 17 15:49:49.825356 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.825331 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2dc23-29827665-vp94k"] Sep 17 15:49:49.838832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.838809 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7"] Sep 17 15:49:49.839558 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.839292 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" podUID="c62ab279-57e6-429d-8adc-403d62adafa1" containerName="payload-processing" containerID="cri-o://b1cb6911644a0270d780efbdaf7f795dcfa8d2283cb4df6b80d56bb2c42d4749" gracePeriod=30 Sep 17 15:49:49.853195 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.853169 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8"] Sep 17 15:49:49.853412 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.853389 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" podUID="568f21c4-ab00-4891-9b22-22dd66d0575d" containerName="payload-pre-processing" containerID="cri-o://1f03b5793c3fef22cfd3bc327257604a02364b46aac23f91d376890f28e67f6e" gracePeriod=30 Sep 17 15:49:49.861324 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.861300 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-77bf5a" Sep 17 15:49:49.914584 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.914539 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:49.988748 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:49.988489 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:50.054620 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.054569 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:50.205035 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.201991 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7788-77bf5a"] Sep 17 15:49:50.215309 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.215283 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" Sep 17 15:49:50.217755 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.217734 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:49:50.220176 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.220157 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" Sep 17 15:49:50.312860 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.312833 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f6762f-ad32-458d-88b3-34ccb6232990-openshift-service-ca\") pod \"57f6762f-ad32-458d-88b3-34ccb6232990\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " Sep 17 15:49:50.313127 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.313107 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d65jt\" (UniqueName: \"kubernetes.io/projected/57f6762f-ad32-458d-88b3-34ccb6232990-kube-api-access-d65jt\") pod \"57f6762f-ad32-458d-88b3-34ccb6232990\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " Sep 17 15:49:50.313234 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.313137 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f6762f-ad32-458d-88b3-34ccb6232990-metrics-certs\") pod \"57f6762f-ad32-458d-88b3-34ccb6232990\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " Sep 17 15:49:50.313234 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.313156 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57f6762f-ad32-458d-88b3-34ccb6232990-maas-api-tls\") pod \"57f6762f-ad32-458d-88b3-34ccb6232990\" (UID: \"57f6762f-ad32-458d-88b3-34ccb6232990\") " Sep 17 15:49:50.313234 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.313181 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62ab279-57e6-429d-8adc-403d62adafa1-plugins-config-volume\") pod \"c62ab279-57e6-429d-8adc-403d62adafa1\" (UID: \"c62ab279-57e6-429d-8adc-403d62adafa1\") " Sep 17 15:49:50.313234 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.313221 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sknks\" (UniqueName: \"kubernetes.io/projected/c62ab279-57e6-429d-8adc-403d62adafa1-kube-api-access-sknks\") pod \"c62ab279-57e6-429d-8adc-403d62adafa1\" (UID: \"c62ab279-57e6-429d-8adc-403d62adafa1\") " Sep 17 15:49:50.313372 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.313256 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f6762f-ad32-458d-88b3-34ccb6232990-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57f6762f-ad32-458d-88b3-34ccb6232990" (UID: "57f6762f-ad32-458d-88b3-34ccb6232990"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:49:50.313437 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.313426 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57f6762f-ad32-458d-88b3-34ccb6232990-openshift-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:50.314027 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.314001 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c62ab279-57e6-429d-8adc-403d62adafa1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c62ab279-57e6-429d-8adc-403d62adafa1" (UID: "c62ab279-57e6-429d-8adc-403d62adafa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:49:50.315396 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.315361 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f6762f-ad32-458d-88b3-34ccb6232990-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "57f6762f-ad32-458d-88b3-34ccb6232990" (UID: "57f6762f-ad32-458d-88b3-34ccb6232990"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:49:50.315708 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.315683 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f6762f-ad32-458d-88b3-34ccb6232990-kube-api-access-d65jt" (OuterVolumeSpecName: "kube-api-access-d65jt") pod "57f6762f-ad32-458d-88b3-34ccb6232990" (UID: "57f6762f-ad32-458d-88b3-34ccb6232990"). InnerVolumeSpecName "kube-api-access-d65jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:49:50.315868 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.315845 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f6762f-ad32-458d-88b3-34ccb6232990-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57f6762f-ad32-458d-88b3-34ccb6232990" (UID: "57f6762f-ad32-458d-88b3-34ccb6232990"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:49:50.316019 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.315998 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62ab279-57e6-429d-8adc-403d62adafa1-kube-api-access-sknks" (OuterVolumeSpecName: "kube-api-access-sknks") pod "c62ab279-57e6-429d-8adc-403d62adafa1" (UID: "c62ab279-57e6-429d-8adc-403d62adafa1"). InnerVolumeSpecName "kube-api-access-sknks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:49:50.394971 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.394805 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:50.414205 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.414176 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568f21c4-ab00-4891-9b22-22dd66d0575d-plugins-config-volume\") pod \"568f21c4-ab00-4891-9b22-22dd66d0575d\" (UID: \"568f21c4-ab00-4891-9b22-22dd66d0575d\") " Sep 17 15:49:50.414337 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.414236 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k95tc\" (UniqueName: \"kubernetes.io/projected/568f21c4-ab00-4891-9b22-22dd66d0575d-kube-api-access-k95tc\") pod \"568f21c4-ab00-4891-9b22-22dd66d0575d\" (UID: \"568f21c4-ab00-4891-9b22-22dd66d0575d\") " Sep 17 15:49:50.414401 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.414389 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d65jt\" (UniqueName: \"kubernetes.io/projected/57f6762f-ad32-458d-88b3-34ccb6232990-kube-api-access-d65jt\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:50.414439 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.414403 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57f6762f-ad32-458d-88b3-34ccb6232990-metrics-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:50.414439 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.414413 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57f6762f-ad32-458d-88b3-34ccb6232990-maas-api-tls\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:50.414439 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.414421 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62ab279-57e6-429d-8adc-403d62adafa1-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:50.414439 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.414430 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sknks\" (UniqueName: \"kubernetes.io/projected/c62ab279-57e6-429d-8adc-403d62adafa1-kube-api-access-sknks\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:50.414572 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.414482 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/568f21c4-ab00-4891-9b22-22dd66d0575d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "568f21c4-ab00-4891-9b22-22dd66d0575d" (UID: "568f21c4-ab00-4891-9b22-22dd66d0575d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:49:50.416235 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.416212 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568f21c4-ab00-4891-9b22-22dd66d0575d-kube-api-access-k95tc" (OuterVolumeSpecName: "kube-api-access-k95tc") pod "568f21c4-ab00-4891-9b22-22dd66d0575d" (UID: "568f21c4-ab00-4891-9b22-22dd66d0575d"). InnerVolumeSpecName "kube-api-access-k95tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:49:50.515079 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.514998 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k95tc\" (UniqueName: \"kubernetes.io/projected/568f21c4-ab00-4891-9b22-22dd66d0575d-kube-api-access-k95tc\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:50.515079 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.515024 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568f21c4-ab00-4891-9b22-22dd66d0575d-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:50.737347 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.737308 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7788-77bf5a" event={"ID":"cfdfed90-bc9b-4295-97b5-225e53185205","Type":"ContainerStarted","Data":"c300f0ca8a835197e31bd9f467ed326c51fc27b3369927022038ba0378147600"} Sep 17 15:49:50.737698 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.737368 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7788-77bf5a" event={"ID":"cfdfed90-bc9b-4295-97b5-225e53185205","Type":"ContainerStarted","Data":"3ea214a19876ed240b573a7801a3e0649b2cd9e8dcfa9ced4aa77995d37ea1b1"} Sep 17 15:49:50.738717 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.738694 2581 generic.go:356] "Generic (PLEG): container finished" podID="c62ab279-57e6-429d-8adc-403d62adafa1" containerID="b1cb6911644a0270d780efbdaf7f795dcfa8d2283cb4df6b80d56bb2c42d4749" exitCode=0 Sep 17 15:49:50.738822 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.738742 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" Sep 17 15:49:50.738822 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.738765 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" event={"ID":"c62ab279-57e6-429d-8adc-403d62adafa1","Type":"ContainerDied","Data":"b1cb6911644a0270d780efbdaf7f795dcfa8d2283cb4df6b80d56bb2c42d4749"} Sep 17 15:49:50.738822 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.738807 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7" event={"ID":"c62ab279-57e6-429d-8adc-403d62adafa1","Type":"ContainerDied","Data":"72d7d3b999fc9e5ce053bfb5bea4ccbee19943fa429d6d07059203f5bea7df07"} Sep 17 15:49:50.738994 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.738837 2581 scope.go:117] "RemoveContainer" containerID="b1cb6911644a0270d780efbdaf7f795dcfa8d2283cb4df6b80d56bb2c42d4749" Sep 17 15:49:50.740056 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.740035 2581 generic.go:356] "Generic (PLEG): container finished" podID="568f21c4-ab00-4891-9b22-22dd66d0575d" containerID="1f03b5793c3fef22cfd3bc327257604a02364b46aac23f91d376890f28e67f6e" exitCode=0 Sep 17 15:49:50.740172 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.740081 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" Sep 17 15:49:50.740172 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.740103 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" event={"ID":"568f21c4-ab00-4891-9b22-22dd66d0575d","Type":"ContainerDied","Data":"1f03b5793c3fef22cfd3bc327257604a02364b46aac23f91d376890f28e67f6e"} Sep 17 15:49:50.740172 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.740142 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8" event={"ID":"568f21c4-ab00-4891-9b22-22dd66d0575d","Type":"ContainerDied","Data":"0591753026e176be8711271b4a827d9f29aae44364d6bc5e49e55e4829d0df0b"} Sep 17 15:49:50.745430 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.745049 2581 generic.go:356] "Generic (PLEG): container finished" podID="57f6762f-ad32-458d-88b3-34ccb6232990" containerID="bc5082140c8433d3f0b6b2e8333f6d802dc07714f7ccb2499cb8d5c0f3897853" exitCode=0 Sep 17 15:49:50.745430 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.745281 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7788-d17a7c" podUID="ebd7686b-b95f-4237-bd2d-38593650a586" containerName="test-curl-7788-d17a7c" containerID="cri-o://1cce9162430f1720b084c97371eaa8f353dda56a196e48080a39ab8750339fec" gracePeriod=2 Sep 17 15:49:50.745588 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.745433 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" Sep 17 15:49:50.745588 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.745470 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" event={"ID":"57f6762f-ad32-458d-88b3-34ccb6232990","Type":"ContainerDied","Data":"bc5082140c8433d3f0b6b2e8333f6d802dc07714f7ccb2499cb8d5c0f3897853"} Sep 17 15:49:50.745588 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.745518 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll" event={"ID":"57f6762f-ad32-458d-88b3-34ccb6232990","Type":"ContainerDied","Data":"a45daa66704bb273843aa11a359f96fc87afe28bec6fb3d6ec44ff2cba5770de"} Sep 17 15:49:50.754177 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.754140 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7788-77bf5a" podStartSLOduration=1.75412604 podStartE2EDuration="1.75412604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:50.752433398 +0000 UTC m=+1177.748670135" watchObservedRunningTime="2026-09-17 15:49:50.75412604 +0000 UTC m=+1177.750362787" Sep 17 15:49:50.786100 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.786078 2581 scope.go:117] "RemoveContainer" containerID="b1cb6911644a0270d780efbdaf7f795dcfa8d2283cb4df6b80d56bb2c42d4749" Sep 17 15:49:50.786465 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:50.786446 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1cb6911644a0270d780efbdaf7f795dcfa8d2283cb4df6b80d56bb2c42d4749\": container with ID starting with b1cb6911644a0270d780efbdaf7f795dcfa8d2283cb4df6b80d56bb2c42d4749 not found: ID does not exist" containerID="b1cb6911644a0270d780efbdaf7f795dcfa8d2283cb4df6b80d56bb2c42d4749" Sep 17 15:49:50.786527 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.786474 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1cb6911644a0270d780efbdaf7f795dcfa8d2283cb4df6b80d56bb2c42d4749"} err="failed to get container status \"b1cb6911644a0270d780efbdaf7f795dcfa8d2283cb4df6b80d56bb2c42d4749\": rpc error: code = NotFound desc = could not find container \"b1cb6911644a0270d780efbdaf7f795dcfa8d2283cb4df6b80d56bb2c42d4749\": container with ID starting with b1cb6911644a0270d780efbdaf7f795dcfa8d2283cb4df6b80d56bb2c42d4749 not found: ID does not exist" Sep 17 15:49:50.786527 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.786492 2581 scope.go:117] "RemoveContainer" containerID="1f03b5793c3fef22cfd3bc327257604a02364b46aac23f91d376890f28e67f6e" Sep 17 15:49:50.789219 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.789193 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 17 15:49:50.789333 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.789255 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 17 15:49:50.791137 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.791115 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7"] Sep 17 15:49:50.795145 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.795113 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a2dc23-75d74fb8d5-4zzs7"] Sep 17 15:49:50.795733 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.795718 2581 scope.go:117] "RemoveContainer" containerID="1f03b5793c3fef22cfd3bc327257604a02364b46aac23f91d376890f28e67f6e" Sep 17 15:49:50.796074 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:50.796046 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f03b5793c3fef22cfd3bc327257604a02364b46aac23f91d376890f28e67f6e\": container with ID starting with 1f03b5793c3fef22cfd3bc327257604a02364b46aac23f91d376890f28e67f6e not found: ID does not exist" containerID="1f03b5793c3fef22cfd3bc327257604a02364b46aac23f91d376890f28e67f6e" Sep 17 15:49:50.796163 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.796080 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f03b5793c3fef22cfd3bc327257604a02364b46aac23f91d376890f28e67f6e"} err="failed to get container status \"1f03b5793c3fef22cfd3bc327257604a02364b46aac23f91d376890f28e67f6e\": rpc error: code = NotFound desc = could not find container \"1f03b5793c3fef22cfd3bc327257604a02364b46aac23f91d376890f28e67f6e\": container with ID starting with 1f03b5793c3fef22cfd3bc327257604a02364b46aac23f91d376890f28e67f6e not found: ID does not exist" Sep 17 15:49:50.796163 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.796097 2581 scope.go:117] "RemoveContainer" containerID="bc5082140c8433d3f0b6b2e8333f6d802dc07714f7ccb2499cb8d5c0f3897853" Sep 17 15:49:50.809619 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.807749 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8"] Sep 17 15:49:50.809619 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.807767 2581 scope.go:117] "RemoveContainer" containerID="bc5082140c8433d3f0b6b2e8333f6d802dc07714f7ccb2499cb8d5c0f3897853" Sep 17 15:49:50.809619 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:50.808120 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5082140c8433d3f0b6b2e8333f6d802dc07714f7ccb2499cb8d5c0f3897853\": container with ID starting with bc5082140c8433d3f0b6b2e8333f6d802dc07714f7ccb2499cb8d5c0f3897853 not found: ID does not exist" containerID="bc5082140c8433d3f0b6b2e8333f6d802dc07714f7ccb2499cb8d5c0f3897853" Sep 17 15:49:50.809619 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.808159 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5082140c8433d3f0b6b2e8333f6d802dc07714f7ccb2499cb8d5c0f3897853"} err="failed to get container status \"bc5082140c8433d3f0b6b2e8333f6d802dc07714f7ccb2499cb8d5c0f3897853\": rpc error: code = NotFound desc = could not find container \"bc5082140c8433d3f0b6b2e8333f6d802dc07714f7ccb2499cb8d5c0f3897853\": container with ID starting with bc5082140c8433d3f0b6b2e8333f6d802dc07714f7ccb2499cb8d5c0f3897853 not found: ID does not exist" Sep 17 15:49:50.812323 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.812288 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2dc23-78cf4988d6-n4dc8"] Sep 17 15:49:50.837369 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.832959 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll"] Sep 17 15:49:50.838130 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.838099 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a2dc23-6fb5dcbdcc-7ptll"] Sep 17 15:49:50.877677 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.877645 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:50.948321 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:50.948295 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-9586b5" Sep 17 15:49:51.029584 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.029558 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fbsf\" (UniqueName: \"kubernetes.io/projected/0a424c04-ef8c-49d7-ae01-2fc346f79013-kube-api-access-7fbsf\") pod \"0a424c04-ef8c-49d7-ae01-2fc346f79013\" (UID: \"0a424c04-ef8c-49d7-ae01-2fc346f79013\") " Sep 17 15:49:51.031382 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.031358 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a424c04-ef8c-49d7-ae01-2fc346f79013-kube-api-access-7fbsf" (OuterVolumeSpecName: "kube-api-access-7fbsf") pod "0a424c04-ef8c-49d7-ae01-2fc346f79013" (UID: "0a424c04-ef8c-49d7-ae01-2fc346f79013"). InnerVolumeSpecName "kube-api-access-7fbsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:49:51.130864 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.130831 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fbsf\" (UniqueName: \"kubernetes.io/projected/0a424c04-ef8c-49d7-ae01-2fc346f79013-kube-api-access-7fbsf\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:51.369213 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.369130 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7788-77bf5a"] Sep 17 15:49:51.372665 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.372640 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7788-77bf5a"] Sep 17 15:49:51.462638 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.462606 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a424c04-ef8c-49d7-ae01-2fc346f79013" path="/var/lib/kubelet/pods/0a424c04-ef8c-49d7-ae01-2fc346f79013/volumes" Sep 17 15:49:51.462981 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.462966 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="568f21c4-ab00-4891-9b22-22dd66d0575d" path="/var/lib/kubelet/pods/568f21c4-ab00-4891-9b22-22dd66d0575d/volumes" Sep 17 15:49:51.463327 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.463314 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57f6762f-ad32-458d-88b3-34ccb6232990" path="/var/lib/kubelet/pods/57f6762f-ad32-458d-88b3-34ccb6232990/volumes" Sep 17 15:49:51.463564 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.463553 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75e3c420-d4c3-4556-9dae-7e14144a28fc" path="/var/lib/kubelet/pods/75e3c420-d4c3-4556-9dae-7e14144a28fc/volumes" Sep 17 15:49:51.464150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.464137 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c62ab279-57e6-429d-8adc-403d62adafa1" path="/var/lib/kubelet/pods/c62ab279-57e6-429d-8adc-403d62adafa1/volumes" Sep 17 15:49:51.491088 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.491061 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7788-bf6860"] Sep 17 15:49:51.491435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.491420 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c62ab279-57e6-429d-8adc-403d62adafa1" containerName="payload-processing" Sep 17 15:49:51.491478 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.491438 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c62ab279-57e6-429d-8adc-403d62adafa1" containerName="payload-processing" Sep 17 15:49:51.491478 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.491445 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfdfed90-bc9b-4295-97b5-225e53185205" containerName="test-curl-7788-77bf5a" Sep 17 15:49:51.491478 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.491452 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfdfed90-bc9b-4295-97b5-225e53185205" containerName="test-curl-7788-77bf5a" Sep 17 15:49:51.491478 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.491471 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f6762f-ad32-458d-88b3-34ccb6232990" containerName="maas-api" Sep 17 15:49:51.491478 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.491478 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f6762f-ad32-458d-88b3-34ccb6232990" containerName="maas-api" Sep 17 15:49:51.491628 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.491491 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="568f21c4-ab00-4891-9b22-22dd66d0575d" containerName="payload-pre-processing" Sep 17 15:49:51.491628 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.491500 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="568f21c4-ab00-4891-9b22-22dd66d0575d" containerName="payload-pre-processing" Sep 17 15:49:51.491628 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.491553 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="568f21c4-ab00-4891-9b22-22dd66d0575d" containerName="payload-pre-processing" Sep 17 15:49:51.491628 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.491562 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfdfed90-bc9b-4295-97b5-225e53185205" containerName="test-curl-7788-77bf5a" Sep 17 15:49:51.491628 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.491572 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57f6762f-ad32-458d-88b3-34ccb6232990" containerName="maas-api" Sep 17 15:49:51.491628 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.491583 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c62ab279-57e6-429d-8adc-403d62adafa1" containerName="payload-processing" Sep 17 15:49:51.495630 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.495614 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-bf6860" Sep 17 15:49:51.502381 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.502356 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7788-bf6860"] Sep 17 15:49:51.604015 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.603984 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:51.635860 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.635780 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcrfw\" (UniqueName: \"kubernetes.io/projected/767b137c-75bc-4749-bc39-3f750f43e348-kube-api-access-kcrfw\") pod \"test-curl-7788-bf6860\" (UID: \"767b137c-75bc-4749-bc39-3f750f43e348\") " pod="opendatahub/test-curl-7788-bf6860" Sep 17 15:49:51.736322 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.736288 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcrfw\" (UniqueName: \"kubernetes.io/projected/767b137c-75bc-4749-bc39-3f750f43e348-kube-api-access-kcrfw\") pod \"test-curl-7788-bf6860\" (UID: \"767b137c-75bc-4749-bc39-3f750f43e348\") " pod="opendatahub/test-curl-7788-bf6860" Sep 17 15:49:51.748260 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.748229 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcrfw\" (UniqueName: \"kubernetes.io/projected/767b137c-75bc-4749-bc39-3f750f43e348-kube-api-access-kcrfw\") pod \"test-curl-7788-bf6860\" (UID: \"767b137c-75bc-4749-bc39-3f750f43e348\") " pod="opendatahub/test-curl-7788-bf6860" Sep 17 15:49:51.752187 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.752161 2581 generic.go:356] "Generic (PLEG): container finished" podID="0a424c04-ef8c-49d7-ae01-2fc346f79013" containerID="75d7e2633d72f3f1ed1e804be179cc16629b996e3ba7eacb4c27a524220519fe" exitCode=137 Sep 17 15:49:51.752314 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.752203 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-9586b5" Sep 17 15:49:51.752314 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.752256 2581 scope.go:117] "RemoveContainer" containerID="75d7e2633d72f3f1ed1e804be179cc16629b996e3ba7eacb4c27a524220519fe" Sep 17 15:49:51.755685 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:51.755663 2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7788-77bf5a" container="test-curl-7788-77bf5a" Sep 17 15:49:51.762024 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.762000 2581 scope.go:117] "RemoveContainer" containerID="75d7e2633d72f3f1ed1e804be179cc16629b996e3ba7eacb4c27a524220519fe" Sep 17 15:49:51.762314 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:51.762292 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d7e2633d72f3f1ed1e804be179cc16629b996e3ba7eacb4c27a524220519fe\": container with ID starting with 75d7e2633d72f3f1ed1e804be179cc16629b996e3ba7eacb4c27a524220519fe not found: ID does not exist" containerID="75d7e2633d72f3f1ed1e804be179cc16629b996e3ba7eacb4c27a524220519fe" Sep 17 15:49:51.762364 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.762325 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d7e2633d72f3f1ed1e804be179cc16629b996e3ba7eacb4c27a524220519fe"} err="failed to get container status \"75d7e2633d72f3f1ed1e804be179cc16629b996e3ba7eacb4c27a524220519fe\": rpc error: code = NotFound desc = could not find container \"75d7e2633d72f3f1ed1e804be179cc16629b996e3ba7eacb4c27a524220519fe\": container with ID starting with 75d7e2633d72f3f1ed1e804be179cc16629b996e3ba7eacb4c27a524220519fe not found: ID does not exist" Sep 17 15:49:51.793661 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.793643 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:51.793939 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.793906 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:51.794600 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.794584 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:49:51.805784 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:51.805770 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-bf6860" Sep 17 15:49:52.080808 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:52.080782 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7788-bf6860"] Sep 17 15:49:52.082487 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:49:52.082458 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod767b137c_75bc_4749_bc39_3f750f43e348.slice/crio-580178f9be5e69772c92ab93669e882a98990eea7ac3e51322e59fbfc670dca1 WatchSource:0}: Error finding container 580178f9be5e69772c92ab93669e882a98990eea7ac3e51322e59fbfc670dca1: Status 404 returned error can't find the container with id 580178f9be5e69772c92ab93669e882a98990eea7ac3e51322e59fbfc670dca1 Sep 17 15:49:52.503118 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:52.503090 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:52.763178 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:52.763148 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7788-bf6860" event={"ID":"767b137c-75bc-4749-bc39-3f750f43e348","Type":"ContainerStarted","Data":"1aa39babee731147e467af4e6882d5af827c477315b54917fff7ce319785e4ea"} Sep 17 15:49:52.763528 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:52.763196 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7788-bf6860" event={"ID":"767b137c-75bc-4749-bc39-3f750f43e348","Type":"ContainerStarted","Data":"580178f9be5e69772c92ab93669e882a98990eea7ac3e51322e59fbfc670dca1"} Sep 17 15:49:52.764406 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:52.764373 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7788-77bf5a" podUID="cfdfed90-bc9b-4295-97b5-225e53185205" containerName="test-curl-7788-77bf5a" containerID="cri-o://c300f0ca8a835197e31bd9f467ed326c51fc27b3369927022038ba0378147600" gracePeriod=2 Sep 17 15:49:52.889346 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:52.889322 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-d17a7c" Sep 17 15:49:52.944196 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:52.944166 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjlcs\" (UniqueName: \"kubernetes.io/projected/ebd7686b-b95f-4237-bd2d-38593650a586-kube-api-access-mjlcs\") pod \"ebd7686b-b95f-4237-bd2d-38593650a586\" (UID: \"ebd7686b-b95f-4237-bd2d-38593650a586\") " Sep 17 15:49:52.946174 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:52.946153 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd7686b-b95f-4237-bd2d-38593650a586-kube-api-access-mjlcs" (OuterVolumeSpecName: "kube-api-access-mjlcs") pod "ebd7686b-b95f-4237-bd2d-38593650a586" (UID: "ebd7686b-b95f-4237-bd2d-38593650a586"). InnerVolumeSpecName "kube-api-access-mjlcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:49:53.044723 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:53.044692 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjlcs\" (UniqueName: \"kubernetes.io/projected/ebd7686b-b95f-4237-bd2d-38593650a586-kube-api-access-mjlcs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:53.461974 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:53.461938 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd7686b-b95f-4237-bd2d-38593650a586" path="/var/lib/kubelet/pods/ebd7686b-b95f-4237-bd2d-38593650a586/volumes" Sep 17 15:49:53.502809 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:53.502769 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:53.541753 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:53.541699 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7788-bf6860" podStartSLOduration=2.541680272 podStartE2EDuration="2.541680272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:52.821666312 +0000 UTC m=+1179.817903060" watchObservedRunningTime="2026-09-17 15:49:53.541680272 +0000 UTC m=+1180.537917017" Sep 17 15:49:53.541953 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:53.541938 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7788-bf6860"] Sep 17 15:49:53.546758 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:53.546693 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7788-bf6860"] Sep 17 15:49:53.770908 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:53.770818 2581 generic.go:356] "Generic (PLEG): container finished" podID="ebd7686b-b95f-4237-bd2d-38593650a586" containerID="1cce9162430f1720b084c97371eaa8f353dda56a196e48080a39ab8750339fec" exitCode=137 Sep 17 15:49:53.770908 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:53.770872 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-d17a7c" Sep 17 15:49:53.771402 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:53.770934 2581 scope.go:117] "RemoveContainer" containerID="1cce9162430f1720b084c97371eaa8f353dda56a196e48080a39ab8750339fec" Sep 17 15:49:53.771402 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:53.771220 2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7788-bf6860" container="test-curl-7788-bf6860" Sep 17 15:49:53.779739 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:53.779697 2581 scope.go:117] "RemoveContainer" containerID="1cce9162430f1720b084c97371eaa8f353dda56a196e48080a39ab8750339fec" Sep 17 15:49:53.780004 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:53.779980 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cce9162430f1720b084c97371eaa8f353dda56a196e48080a39ab8750339fec\": container with ID starting with 1cce9162430f1720b084c97371eaa8f353dda56a196e48080a39ab8750339fec not found: ID does not exist" containerID="1cce9162430f1720b084c97371eaa8f353dda56a196e48080a39ab8750339fec" Sep 17 15:49:53.780092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:53.780014 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cce9162430f1720b084c97371eaa8f353dda56a196e48080a39ab8750339fec"} err="failed to get container status \"1cce9162430f1720b084c97371eaa8f353dda56a196e48080a39ab8750339fec\": rpc error: code = NotFound desc = could not find container \"1cce9162430f1720b084c97371eaa8f353dda56a196e48080a39ab8750339fec\": container with ID starting with 1cce9162430f1720b084c97371eaa8f353dda56a196e48080a39ab8750339fec not found: ID does not exist" Sep 17 15:49:54.218852 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.218815 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:54.539711 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.539680 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89"] Sep 17 15:49:54.540112 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.540097 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="767b137c-75bc-4749-bc39-3f750f43e348" containerName="test-curl-7788-bf6860" Sep 17 15:49:54.540168 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.540113 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="767b137c-75bc-4749-bc39-3f750f43e348" containerName="test-curl-7788-bf6860" Sep 17 15:49:54.540202 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.540188 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="767b137c-75bc-4749-bc39-3f750f43e348" containerName="test-curl-7788-bf6860" Sep 17 15:49:54.544232 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.544216 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.547654 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.547629 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-99719fde-openshift-default-dockercfg-lwxbs\"" Sep 17 15:49:54.554688 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.554671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-istio-envoy\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.554788 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.554704 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-istio-data\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.554788 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.554729 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxhw7\" (UniqueName: \"kubernetes.io/projected/452e087d-09a4-467c-ae12-a1b110042ab2-kube-api-access-jxhw7\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.554788 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.554781 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452e087d-09a4-467c-ae12-a1b110042ab2-istio-token\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.554904 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.554802 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452e087d-09a4-467c-ae12-a1b110042ab2-istiod-ca-cert\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.554904 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.554818 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-credential-socket\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.555001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.554902 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-workload-certs\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.555001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.554954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-workload-socket\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.555001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.554985 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452e087d-09a4-467c-ae12-a1b110042ab2-istio-podinfo\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.562315 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.562291 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89"] Sep 17 15:49:54.582056 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.582026 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:54.656168 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.656139 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452e087d-09a4-467c-ae12-a1b110042ab2-istio-podinfo\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.656359 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.656173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-istio-envoy\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.656359 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.656205 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-istio-data\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.656359 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.656230 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxhw7\" (UniqueName: \"kubernetes.io/projected/452e087d-09a4-467c-ae12-a1b110042ab2-kube-api-access-jxhw7\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.656359 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.656265 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452e087d-09a4-467c-ae12-a1b110042ab2-istio-token\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.656359 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.656287 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452e087d-09a4-467c-ae12-a1b110042ab2-istiod-ca-cert\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.656359 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.656304 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-credential-socket\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.656359 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.656342 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-workload-certs\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.656740 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.656378 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-workload-socket\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.656740 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.656621 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-istio-data\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.656740 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.656709 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-workload-socket\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.656896 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.656769 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-credential-socket\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.656896 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.656804 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-workload-certs\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.657236 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.657215 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452e087d-09a4-467c-ae12-a1b110042ab2-istiod-ca-cert\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.658447 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.658429 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-istio-envoy\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.658648 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.658633 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452e087d-09a4-467c-ae12-a1b110042ab2-istio-podinfo\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.669729 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.669710 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452e087d-09a4-467c-ae12-a1b110042ab2-istio-token\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.673809 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.673790 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxhw7\" (UniqueName: \"kubernetes.io/projected/452e087d-09a4-467c-ae12-a1b110042ab2-kube-api-access-jxhw7\") pod \"e2e-mt-99719fde-openshift-default-595b55748-pjw89\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.778849 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.778820 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7788-bf6860" podUID="767b137c-75bc-4749-bc39-3f750f43e348" containerName="test-curl-7788-bf6860" containerID="cri-o://1aa39babee731147e467af4e6882d5af827c477315b54917fff7ce319785e4ea" gracePeriod=2 Sep 17 15:49:54.780962 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:54.780933 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7788-bf6860\" is forbidden: User \"system:node:ip-10-0-128-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-15.ec2.internal' and this object" podUID="767b137c-75bc-4749-bc39-3f750f43e348" pod="opendatahub/test-curl-7788-bf6860" Sep 17 15:49:54.855077 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.855044 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:54.860757 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.860732 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:54.914578 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.914548 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-77bf5a" Sep 17 15:49:54.917142 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:54.917116 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7788-bf6860\" is forbidden: User \"system:node:ip-10-0-128-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-15.ec2.internal' and this object" podUID="767b137c-75bc-4749-bc39-3f750f43e348" pod="opendatahub/test-curl-7788-bf6860" Sep 17 15:49:54.958489 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.958458 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn26g\" (UniqueName: \"kubernetes.io/projected/cfdfed90-bc9b-4295-97b5-225e53185205-kube-api-access-bn26g\") pod \"cfdfed90-bc9b-4295-97b5-225e53185205\" (UID: \"cfdfed90-bc9b-4295-97b5-225e53185205\") " Sep 17 15:49:54.960330 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:54.960308 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfdfed90-bc9b-4295-97b5-225e53185205-kube-api-access-bn26g" (OuterVolumeSpecName: "kube-api-access-bn26g") pod "cfdfed90-bc9b-4295-97b5-225e53185205" (UID: "cfdfed90-bc9b-4295-97b5-225e53185205"). InnerVolumeSpecName "kube-api-access-bn26g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:49:55.060132 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.060064 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn26g\" (UniqueName: \"kubernetes.io/projected/cfdfed90-bc9b-4295-97b5-225e53185205-kube-api-access-bn26g\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:55.140583 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:49:55.140558 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452e087d_09a4_467c_ae12_a1b110042ab2.slice/crio-f1ad19921feb49dd9ad8406d8ec489e768b2dc916065e24347a72f59bd5fddc4 WatchSource:0}: Error finding container f1ad19921feb49dd9ad8406d8ec489e768b2dc916065e24347a72f59bd5fddc4: Status 404 returned error can't find the container with id f1ad19921feb49dd9ad8406d8ec489e768b2dc916065e24347a72f59bd5fddc4 Sep 17 15:49:55.141716 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.141689 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89"] Sep 17 15:49:55.142888 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.142856 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:49:55.143000 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.142962 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:49:55.143074 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.143013 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:49:55.462806 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.462726 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfdfed90-bc9b-4295-97b5-225e53185205" path="/var/lib/kubelet/pods/cfdfed90-bc9b-4295-97b5-225e53185205/volumes" Sep 17 15:49:55.785225 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.785188 2581 generic.go:356] "Generic (PLEG): container finished" podID="cfdfed90-bc9b-4295-97b5-225e53185205" containerID="c300f0ca8a835197e31bd9f467ed326c51fc27b3369927022038ba0378147600" exitCode=137 Sep 17 15:49:55.785629 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.785232 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-77bf5a" Sep 17 15:49:55.785629 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.785265 2581 scope.go:117] "RemoveContainer" containerID="c300f0ca8a835197e31bd9f467ed326c51fc27b3369927022038ba0378147600" Sep 17 15:49:55.787147 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.787119 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" event={"ID":"452e087d-09a4-467c-ae12-a1b110042ab2","Type":"ContainerStarted","Data":"bebb145ca75fbdc5904f1c10ea4696439efb54135cba250783d895911fd3dcf0"} Sep 17 15:49:55.787272 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.787175 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" event={"ID":"452e087d-09a4-467c-ae12-a1b110042ab2","Type":"ContainerStarted","Data":"f1ad19921feb49dd9ad8406d8ec489e768b2dc916065e24347a72f59bd5fddc4"} Sep 17 15:49:55.794284 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.794265 2581 scope.go:117] "RemoveContainer" containerID="c300f0ca8a835197e31bd9f467ed326c51fc27b3369927022038ba0378147600" Sep 17 15:49:55.794543 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:55.794520 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c300f0ca8a835197e31bd9f467ed326c51fc27b3369927022038ba0378147600\": container with ID starting with c300f0ca8a835197e31bd9f467ed326c51fc27b3369927022038ba0378147600 not found: ID does not exist" containerID="c300f0ca8a835197e31bd9f467ed326c51fc27b3369927022038ba0378147600" Sep 17 15:49:55.794611 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.794550 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c300f0ca8a835197e31bd9f467ed326c51fc27b3369927022038ba0378147600"} err="failed to get container status \"c300f0ca8a835197e31bd9f467ed326c51fc27b3369927022038ba0378147600\": rpc error: code = NotFound desc = could not find container \"c300f0ca8a835197e31bd9f467ed326c51fc27b3369927022038ba0378147600\": container with ID starting with c300f0ca8a835197e31bd9f467ed326c51fc27b3369927022038ba0378147600 not found: ID does not exist" Sep 17 15:49:55.819662 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.819610 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" podStartSLOduration=1.819592039 podStartE2EDuration="1.819592039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:55.816356341 +0000 UTC m=+1182.812593088" watchObservedRunningTime="2026-09-17 15:49:55.819592039 +0000 UTC m=+1182.815828785" Sep 17 15:49:55.856028 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:55.856002 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:56.861597 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:56.861568 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:56.959599 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:56.959578 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-bf6860" Sep 17 15:49:56.977765 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:56.977733 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcrfw\" (UniqueName: \"kubernetes.io/projected/767b137c-75bc-4749-bc39-3f750f43e348-kube-api-access-kcrfw\") pod \"767b137c-75bc-4749-bc39-3f750f43e348\" (UID: \"767b137c-75bc-4749-bc39-3f750f43e348\") " Sep 17 15:49:56.979647 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:56.979624 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767b137c-75bc-4749-bc39-3f750f43e348-kube-api-access-kcrfw" (OuterVolumeSpecName: "kube-api-access-kcrfw") pod "767b137c-75bc-4749-bc39-3f750f43e348" (UID: "767b137c-75bc-4749-bc39-3f750f43e348"). InnerVolumeSpecName "kube-api-access-kcrfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:49:56.993239 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:56.993215 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:57.078713 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.078639 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcrfw\" (UniqueName: \"kubernetes.io/projected/767b137c-75bc-4749-bc39-3f750f43e348-kube-api-access-kcrfw\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:49:57.106614 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.106581 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:57.205829 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.205783 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:57.405181 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.405107 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:57.463406 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.463372 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="767b137c-75bc-4749-bc39-3f750f43e348" path="/var/lib/kubelet/pods/767b137c-75bc-4749-bc39-3f750f43e348/volumes" Sep 17 15:49:57.778725 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.778696 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:57.801069 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.801036 2581 generic.go:356] "Generic (PLEG): container finished" podID="767b137c-75bc-4749-bc39-3f750f43e348" containerID="1aa39babee731147e467af4e6882d5af827c477315b54917fff7ce319785e4ea" exitCode=137 Sep 17 15:49:57.801206 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.801078 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7788-bf6860" Sep 17 15:49:57.801206 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.801163 2581 scope.go:117] "RemoveContainer" containerID="1aa39babee731147e467af4e6882d5af827c477315b54917fff7ce319785e4ea" Sep 17 15:49:57.801447 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.801384 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:57.802669 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.802650 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:49:57.810016 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.809989 2581 scope.go:117] "RemoveContainer" containerID="1aa39babee731147e467af4e6882d5af827c477315b54917fff7ce319785e4ea" Sep 17 15:49:57.810293 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:49:57.810275 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa39babee731147e467af4e6882d5af827c477315b54917fff7ce319785e4ea\": container with ID starting with 1aa39babee731147e467af4e6882d5af827c477315b54917fff7ce319785e4ea not found: ID does not exist" containerID="1aa39babee731147e467af4e6882d5af827c477315b54917fff7ce319785e4ea" Sep 17 15:49:57.810346 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.810301 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa39babee731147e467af4e6882d5af827c477315b54917fff7ce319785e4ea"} err="failed to get container status \"1aa39babee731147e467af4e6882d5af827c477315b54917fff7ce319785e4ea\": rpc error: code = NotFound desc = could not find container \"1aa39babee731147e467af4e6882d5af827c477315b54917fff7ce319785e4ea\": container with ID starting with 1aa39babee731147e467af4e6882d5af827c477315b54917fff7ce319785e4ea not found: ID does not exist" Sep 17 15:49:57.898887 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.898852 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:57.993466 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:57.993430 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:58.095428 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.095345 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:58.329386 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.329348 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44"] Sep 17 15:49:58.334536 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.334512 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.337411 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.337391 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1d63c0-openshift-default-dockercfg-mf9ch\"" Sep 17 15:49:58.352479 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.352428 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44"] Sep 17 15:49:58.360384 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.360364 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:58.391500 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.391470 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58f0775-120c-45da-8eea-9f62402ca713-istiod-ca-cert\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.391627 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.391504 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-istio-envoy\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.391627 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.391532 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-workload-certs\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.391627 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.391597 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58f0775-120c-45da-8eea-9f62402ca713-istio-token\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.391627 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.391619 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-workload-socket\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.391790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.391633 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58f0775-120c-45da-8eea-9f62402ca713-istio-podinfo\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.391790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.391649 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-istio-data\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.391790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.391686 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2crt\" (UniqueName: \"kubernetes.io/projected/e58f0775-120c-45da-8eea-9f62402ca713-kube-api-access-l2crt\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.391790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.391763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-credential-socket\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.492202 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.492173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58f0775-120c-45da-8eea-9f62402ca713-istiod-ca-cert\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.492202 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.492206 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-istio-envoy\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.492441 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.492237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-workload-certs\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.492441 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.492293 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58f0775-120c-45da-8eea-9f62402ca713-istio-token\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.492441 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.492328 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-workload-socket\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.492441 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.492351 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58f0775-120c-45da-8eea-9f62402ca713-istio-podinfo\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.492441 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.492376 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-istio-data\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.492759 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.492733 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-istio-data\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.492825 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.492752 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-workload-certs\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.492825 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.492784 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-workload-socket\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.493016 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.492746 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2crt\" (UniqueName: \"kubernetes.io/projected/e58f0775-120c-45da-8eea-9f62402ca713-kube-api-access-l2crt\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.493122 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.493065 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58f0775-120c-45da-8eea-9f62402ca713-istiod-ca-cert\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.493122 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.493077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-credential-socket\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.493363 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.493343 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-credential-socket\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.494635 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.494616 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-istio-envoy\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.494961 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.494940 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58f0775-120c-45da-8eea-9f62402ca713-istio-podinfo\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.500601 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.500582 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58f0775-120c-45da-8eea-9f62402ca713-istio-token\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.504442 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.504422 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2crt\" (UniqueName: \"kubernetes.io/projected/e58f0775-120c-45da-8eea-9f62402ca713-kube-api-access-l2crt\") pod \"e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.605042 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.604959 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:58.644959 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.644906 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:49:58.925289 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.925263 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44"] Sep 17 15:49:58.926671 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:49:58.926649 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58f0775_120c_45da_8eea_9f62402ca713.slice/crio-bf97190dace79a1f6859d92d64436b20c0b8a51310cd6202b6e21dee2ecc1196 WatchSource:0}: Error finding container bf97190dace79a1f6859d92d64436b20c0b8a51310cd6202b6e21dee2ecc1196: Status 404 returned error can't find the container with id bf97190dace79a1f6859d92d64436b20c0b8a51310cd6202b6e21dee2ecc1196 Sep 17 15:49:58.929058 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.929029 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:49:58.929134 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.929094 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:49:58.929134 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.929121 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:49:58.967505 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:58.967475 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:59.688777 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:59.688738 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:49:59.814934 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:59.814877 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" event={"ID":"e58f0775-120c-45da-8eea-9f62402ca713","Type":"ContainerStarted","Data":"4ccb41635b24fe94727e98d84a2bcd948129a08d80b9ee01023e5c7f8926ff56"} Sep 17 15:49:59.815097 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:59.814968 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" event={"ID":"e58f0775-120c-45da-8eea-9f62402ca713","Type":"ContainerStarted","Data":"bf97190dace79a1f6859d92d64436b20c0b8a51310cd6202b6e21dee2ecc1196"} Sep 17 15:49:59.854219 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:49:59.854170 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" podStartSLOduration=1.854153464 podStartE2EDuration="1.854153464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:59.85262628 +0000 UTC m=+1186.848863041" watchObservedRunningTime="2026-09-17 15:49:59.854153464 +0000 UTC m=+1186.850390210" Sep 17 15:50:00.613032 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:00.613001 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:00.645999 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:00.645972 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:50:00.651362 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:00.651336 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:50:00.821220 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:00.821189 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:50:00.822178 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:00.822157 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:50:01.349636 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.349596 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk"] Sep 17 15:50:01.353446 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.353422 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:01.356400 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.356367 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-99719fde-serving-cert\"" Sep 17 15:50:01.364277 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.364256 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk"] Sep 17 15:50:01.404250 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.404219 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:01.420630 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.420603 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-metrics-certs\") pod \"maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:01.420744 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.420661 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlzxr\" (UniqueName: \"kubernetes.io/projected/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-kube-api-access-zlzxr\") pod \"maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:01.420744 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.420715 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-openshift-service-ca\") pod \"maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:01.420744 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.420736 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-maas-api-tls\") pod \"maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:01.522094 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.522057 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-metrics-certs\") pod \"maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:01.522271 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.522237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlzxr\" (UniqueName: \"kubernetes.io/projected/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-kube-api-access-zlzxr\") pod \"maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:01.522391 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.522372 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-openshift-service-ca\") pod \"maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:01.522449 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.522411 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-maas-api-tls\") pod \"maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:01.522571 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:50:01.522540 2581 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-serving-cert: secret "maas-api-e2e-mt-99719fde-serving-cert" not found Sep 17 15:50:01.522680 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:50:01.522622 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-maas-api-tls podName:ff1e517f-cc2e-42a2-a157-31ee8a0aeeda nodeName:}" failed. No retries permitted until 2026-09-17 15:50:02.022601546 +0000 UTC m=+1189.018838286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-maas-api-tls") pod "maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" (UID: "ff1e517f-cc2e-42a2-a157-31ee8a0aeeda") : secret "maas-api-e2e-mt-99719fde-serving-cert" not found Sep 17 15:50:01.523046 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.523027 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-openshift-service-ca\") pod \"maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:01.524528 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.524509 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-metrics-certs\") pod \"maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:01.530586 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.530561 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:01.532317 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.532294 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlzxr\" (UniqueName: \"kubernetes.io/projected/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-kube-api-access-zlzxr\") pod \"maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:01.883878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:01.883845 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:02.027462 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:02.027399 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-maas-api-tls\") pod \"maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:02.029868 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:02.029845 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-maas-api-tls\") pod \"maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:02.080756 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:02.080701 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:02.194127 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:02.194025 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:02.266454 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:02.266418 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:02.569194 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:02.569167 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk"] Sep 17 15:50:02.570716 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:50:02.570687 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1e517f_cc2e_42a2_a157_31ee8a0aeeda.slice/crio-e6aafbff1445fcf8113f611ff5f2563f82d0d6165809df9a99345b1e6ee3fb41 WatchSource:0}: Error finding container e6aafbff1445fcf8113f611ff5f2563f82d0d6165809df9a99345b1e6ee3fb41: Status 404 returned error can't find the container with id e6aafbff1445fcf8113f611ff5f2563f82d0d6165809df9a99345b1e6ee3fb41 Sep 17 15:50:02.572171 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:02.572156 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 15:50:02.698297 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:02.698263 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:02.833246 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:02.833157 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" event={"ID":"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda","Type":"ContainerStarted","Data":"e6aafbff1445fcf8113f611ff5f2563f82d0d6165809df9a99345b1e6ee3fb41"} Sep 17 15:50:03.580448 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:03.580414 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:03.839835 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:03.839799 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" event={"ID":"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda","Type":"ContainerStarted","Data":"f1e9e0f94cac800c3befe7f2fd58e8a10e38ada784bb3977584408604e4c541b"} Sep 17 15:50:03.840042 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:03.839945 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:03.872254 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:03.872205 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" podStartSLOduration=2.04766102 podStartE2EDuration="2.872190974s" totalImagesPullingTime="824.529954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:03.870363944 +0000 UTC m=+1190.866600691" watchObservedRunningTime="2026-09-17 15:50:03.872190974 +0000 UTC m=+1190.868427756" Sep 17 15:50:03.994105 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:03.994076 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:04.523385 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:04.523351 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:05.619973 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:05.619939 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:07.936642 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:07.936612 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:08.282085 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.282055 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:08.383993 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.383962 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:08.525253 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.525223 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:08.777192 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.777150 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:08.879229 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.879198 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:08.906317 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.906284 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4"] Sep 17 15:50:08.909958 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.909936 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" Sep 17 15:50:08.915376 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.915357 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1d63c0-dockercfg-ccc9d\"" Sep 17 15:50:08.915568 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.915550 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1d63c0\"" Sep 17 15:50:08.933716 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.933688 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk"] Sep 17 15:50:08.937801 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.937778 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4"] Sep 17 15:50:08.938097 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.937882 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" Sep 17 15:50:08.950721 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.950700 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk"] Sep 17 15:50:08.990952 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.990904 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4\" (UID: \"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" Sep 17 15:50:08.991079 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.991021 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr5hh\" (UniqueName: \"kubernetes.io/projected/1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33-kube-api-access-mr5hh\") pod \"payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4\" (UID: \"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" Sep 17 15:50:08.991079 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.991047 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7e6cbf-59da-4d79-9623-769a0c5371d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk\" (UID: \"4f7e6cbf-59da-4d79-9623-769a0c5371d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" Sep 17 15:50:08.991159 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:08.991080 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnzv4\" (UniqueName: \"kubernetes.io/projected/4f7e6cbf-59da-4d79-9623-769a0c5371d2-kube-api-access-vnzv4\") pod \"payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk\" (UID: \"4f7e6cbf-59da-4d79-9623-769a0c5371d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" Sep 17 15:50:09.092079 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.091997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4\" (UID: \"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" Sep 17 15:50:09.092079 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.092060 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr5hh\" (UniqueName: \"kubernetes.io/projected/1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33-kube-api-access-mr5hh\") pod \"payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4\" (UID: \"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" Sep 17 15:50:09.092261 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.092079 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7e6cbf-59da-4d79-9623-769a0c5371d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk\" (UID: \"4f7e6cbf-59da-4d79-9623-769a0c5371d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" Sep 17 15:50:09.092261 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.092099 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnzv4\" (UniqueName: \"kubernetes.io/projected/4f7e6cbf-59da-4d79-9623-769a0c5371d2-kube-api-access-vnzv4\") pod \"payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk\" (UID: \"4f7e6cbf-59da-4d79-9623-769a0c5371d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" Sep 17 15:50:09.092815 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.092796 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4\" (UID: \"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" Sep 17 15:50:09.092855 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.092809 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7e6cbf-59da-4d79-9623-769a0c5371d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk\" (UID: \"4f7e6cbf-59da-4d79-9623-769a0c5371d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" Sep 17 15:50:09.103261 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.103232 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnzv4\" (UniqueName: \"kubernetes.io/projected/4f7e6cbf-59da-4d79-9623-769a0c5371d2-kube-api-access-vnzv4\") pod \"payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk\" (UID: \"4f7e6cbf-59da-4d79-9623-769a0c5371d2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" Sep 17 15:50:09.103485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.103469 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr5hh\" (UniqueName: \"kubernetes.io/projected/1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33-kube-api-access-mr5hh\") pod \"payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4\" (UID: \"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" Sep 17 15:50:09.220120 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.220089 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" Sep 17 15:50:09.248673 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.248644 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" Sep 17 15:50:09.527174 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.527146 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4"] Sep 17 15:50:09.537313 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:50:09.537275 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc172b4_e91f_4cbc_bcd0_d0da7e3acd33.slice/crio-16df0037b746289d36a4cb6867fb2e9d5a03e1c6ff000b3d46d7ba928b13b385 WatchSource:0}: Error finding container 16df0037b746289d36a4cb6867fb2e9d5a03e1c6ff000b3d46d7ba928b13b385: Status 404 returned error can't find the container with id 16df0037b746289d36a4cb6867fb2e9d5a03e1c6ff000b3d46d7ba928b13b385 Sep 17 15:50:09.562656 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.562634 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk"] Sep 17 15:50:09.569027 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:50:09.568996 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7e6cbf_59da_4d79_9623_769a0c5371d2.slice/crio-0cb6907376455fc3ea6a99bd8cbf2a111abc74dfe3bad6a4f9bbf31de572cddf WatchSource:0}: Error finding container 0cb6907376455fc3ea6a99bd8cbf2a111abc74dfe3bad6a4f9bbf31de572cddf: Status 404 returned error can't find the container with id 0cb6907376455fc3ea6a99bd8cbf2a111abc74dfe3bad6a4f9bbf31de572cddf Sep 17 15:50:09.851144 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.851059 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:50:09.873451 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.873419 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" event={"ID":"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33","Type":"ContainerStarted","Data":"2d3f8377491b962af286c8b360627f44537d7788795e9cdab4f401dfb5118dd0"} Sep 17 15:50:09.873651 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.873474 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" event={"ID":"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33","Type":"ContainerStarted","Data":"16df0037b746289d36a4cb6867fb2e9d5a03e1c6ff000b3d46d7ba928b13b385"} Sep 17 15:50:09.873651 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.873517 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" Sep 17 15:50:09.875345 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.875315 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" event={"ID":"4f7e6cbf-59da-4d79-9623-769a0c5371d2","Type":"ContainerStarted","Data":"0f0f8f3554e35a89afa47f55d96fa31ba9a0b09d53d5319b894397227dce4d5b"} Sep 17 15:50:09.875478 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.875361 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" event={"ID":"4f7e6cbf-59da-4d79-9623-769a0c5371d2","Type":"ContainerStarted","Data":"0cb6907376455fc3ea6a99bd8cbf2a111abc74dfe3bad6a4f9bbf31de572cddf"} Sep 17 15:50:09.875478 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.875393 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" Sep 17 15:50:09.909089 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.909040 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" podStartSLOduration=1.909024839 podStartE2EDuration="1.909024839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:09.907835512 +0000 UTC m=+1196.904072260" watchObservedRunningTime="2026-09-17 15:50:09.909024839 +0000 UTC m=+1196.905261617" Sep 17 15:50:09.958614 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:09.958565 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" podStartSLOduration=1.958548417 podStartE2EDuration="1.958548417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:09.956511758 +0000 UTC m=+1196.952748506" watchObservedRunningTime="2026-09-17 15:50:09.958548417 +0000 UTC m=+1196.954785160" Sep 17 15:50:20.891143 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:20.891113 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" Sep 17 15:50:20.891640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:20.891199 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" Sep 17 15:50:21.960945 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:21.960892 2581 generic.go:356] "Generic (PLEG): container finished" podID="452e087d-09a4-467c-ae12-a1b110042ab2" containerID="bebb145ca75fbdc5904f1c10ea4696439efb54135cba250783d895911fd3dcf0" exitCode=0 Sep 17 15:50:21.960945 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:21.960942 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" event={"ID":"452e087d-09a4-467c-ae12-a1b110042ab2","Type":"ContainerDied","Data":"bebb145ca75fbdc5904f1c10ea4696439efb54135cba250783d895911fd3dcf0"} Sep 17 15:50:21.961400 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:21.961345 2581 scope.go:117] "RemoveContainer" containerID="bebb145ca75fbdc5904f1c10ea4696439efb54135cba250783d895911fd3dcf0" Sep 17 15:50:21.962544 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:21.962508 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:50:21.962618 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:21.962581 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:50:21.962658 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:21.962632 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:50:22.580718 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:22.580685 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:22.968946 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:22.968846 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" event={"ID":"452e087d-09a4-467c-ae12-a1b110042ab2","Type":"ContainerStarted","Data":"4b7f0fa1a43c7a2db90a9da921b33df9d38b5fdd7b3fa7f4b0491fec1f5fdb80"} Sep 17 15:50:23.856221 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:23.856185 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:50:23.857378 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:23.857351 2581 patch_prober.go:28] interesting pod/e2e-mt-99719fde-openshift-default-595b55748-pjw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 17 15:50:23.857483 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:23.857400 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" podUID="452e087d-09a4-467c-ae12-a1b110042ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 17 15:50:24.615121 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:24.615086 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:24.856242 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:24.856203 2581 patch_prober.go:28] interesting pod/e2e-mt-99719fde-openshift-default-595b55748-pjw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 17 15:50:24.856516 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:24.856257 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" podUID="452e087d-09a4-467c-ae12-a1b110042ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 17 15:50:25.450410 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:25.450330 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:25.855430 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:25.855392 2581 patch_prober.go:28] interesting pod/e2e-mt-99719fde-openshift-default-595b55748-pjw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 17 15:50:25.855890 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:25.855467 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" podUID="452e087d-09a4-467c-ae12-a1b110042ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 17 15:50:26.223229 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:26.223142 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:26.855414 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:26.855378 2581 patch_prober.go:28] interesting pod/e2e-mt-99719fde-openshift-default-595b55748-pjw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 17 15:50:26.855595 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:26.855452 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" podUID="452e087d-09a4-467c-ae12-a1b110042ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 17 15:50:27.107541 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:27.107454 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:27.801384 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:27.801342 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:50:27.856140 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:27.856101 2581 patch_prober.go:28] interesting pod/e2e-mt-99719fde-openshift-default-595b55748-pjw89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 17 15:50:27.856628 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:27.856161 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" podUID="452e087d-09a4-467c-ae12-a1b110042ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 17 15:50:27.947934 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:27.947880 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:28.297377 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:28.297348 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:28.859421 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:28.859389 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:50:28.860366 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:28.860342 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:50:29.212864 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:29.212781 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:30.213838 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:30.213804 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:31.093773 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:31.093673 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:31.827243 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:31.827213 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:31.943647 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:31.943606 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:32.285762 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:32.285733 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:32.764735 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:32.764645 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:32.885017 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:32.884958 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:33.044051 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:33.044007 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:34.397211 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:34.397162 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:35.965772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:35.965727 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:36.857774 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:36.857735 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:37.065601 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:37.065570 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:37.573664 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:37.573632 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:38.334116 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:38.334088 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:39.074537 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:39.074507 2581 generic.go:356] "Generic (PLEG): container finished" podID="e58f0775-120c-45da-8eea-9f62402ca713" containerID="4ccb41635b24fe94727e98d84a2bcd948129a08d80b9ee01023e5c7f8926ff56" exitCode=0 Sep 17 15:50:39.074726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:39.074552 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" event={"ID":"e58f0775-120c-45da-8eea-9f62402ca713","Type":"ContainerDied","Data":"4ccb41635b24fe94727e98d84a2bcd948129a08d80b9ee01023e5c7f8926ff56"} Sep 17 15:50:39.074993 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:39.074971 2581 scope.go:117] "RemoveContainer" containerID="4ccb41635b24fe94727e98d84a2bcd948129a08d80b9ee01023e5c7f8926ff56" Sep 17 15:50:39.076206 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:39.076181 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:50:39.076272 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:39.076250 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:50:39.076314 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:39.076278 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:50:39.231072 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:39.231044 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:40.087206 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:40.087166 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" event={"ID":"e58f0775-120c-45da-8eea-9f62402ca713","Type":"ContainerStarted","Data":"fc8f8a0dd641a42177387b11860fb5c2c30d2a0b6ef205d4beeecaeb759ce765"} Sep 17 15:50:40.257262 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:40.257227 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:40.645990 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:40.645958 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:50:40.646218 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:40.646192 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 17 15:50:40.646265 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:40.646245 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" podUID="e58f0775-120c-45da-8eea-9f62402ca713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 17 15:50:41.119360 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:41.119327 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:41.645518 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:41.645484 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 17 15:50:41.645684 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:41.645542 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" podUID="e58f0775-120c-45da-8eea-9f62402ca713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 17 15:50:42.103271 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:42.103238 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:42.645655 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:42.645620 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 17 15:50:42.646040 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:42.645677 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" podUID="e58f0775-120c-45da-8eea-9f62402ca713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 17 15:50:42.916360 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:42.916279 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:43.017084 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.017055 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:43.531338 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.531300 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:43.624565 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.624533 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:43.645666 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.645631 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 17 15:50:43.646082 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.645698 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" podUID="e58f0775-120c-45da-8eea-9f62402ca713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 17 15:50:43.718149 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.718112 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6"] Sep 17 15:50:43.720743 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.720721 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" Sep 17 15:50:43.723108 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.723086 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-5818c9-dockercfg-8qwnx\"" Sep 17 15:50:43.723320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.723299 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5818c9\"" Sep 17 15:50:43.733653 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.733629 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6"] Sep 17 15:50:43.751204 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.751170 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c28f42c-b413-4a42-98e6-8759f7f4322b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6\" (UID: \"4c28f42c-b413-4a42-98e6-8759f7f4322b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" Sep 17 15:50:43.751376 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.751218 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlh5s\" (UniqueName: \"kubernetes.io/projected/4c28f42c-b413-4a42-98e6-8759f7f4322b-kube-api-access-dlh5s\") pod \"payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6\" (UID: \"4c28f42c-b413-4a42-98e6-8759f7f4322b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" Sep 17 15:50:43.852414 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.852331 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c28f42c-b413-4a42-98e6-8759f7f4322b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6\" (UID: \"4c28f42c-b413-4a42-98e6-8759f7f4322b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" Sep 17 15:50:43.852414 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.852366 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlh5s\" (UniqueName: \"kubernetes.io/projected/4c28f42c-b413-4a42-98e6-8759f7f4322b-kube-api-access-dlh5s\") pod \"payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6\" (UID: \"4c28f42c-b413-4a42-98e6-8759f7f4322b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" Sep 17 15:50:43.853144 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.853124 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c28f42c-b413-4a42-98e6-8759f7f4322b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6\" (UID: \"4c28f42c-b413-4a42-98e6-8759f7f4322b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" Sep 17 15:50:43.862477 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:43.862451 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlh5s\" (UniqueName: \"kubernetes.io/projected/4c28f42c-b413-4a42-98e6-8759f7f4322b-kube-api-access-dlh5s\") pod \"payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6\" (UID: \"4c28f42c-b413-4a42-98e6-8759f7f4322b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" Sep 17 15:50:44.034931 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:44.034892 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" Sep 17 15:50:44.335831 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:44.335807 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6"] Sep 17 15:50:44.337692 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:50:44.337667 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c28f42c_b413_4a42_98e6_8759f7f4322b.slice/crio-9f20ac032c258b473d9f8dbaab06e62128475c184cdf6fb4d96aa047f71cb1f9 WatchSource:0}: Error finding container 9f20ac032c258b473d9f8dbaab06e62128475c184cdf6fb4d96aa047f71cb1f9: Status 404 returned error can't find the container with id 9f20ac032c258b473d9f8dbaab06e62128475c184cdf6fb4d96aa047f71cb1f9 Sep 17 15:50:44.645366 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:44.645286 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 17 15:50:44.645366 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:44.645342 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" podUID="e58f0775-120c-45da-8eea-9f62402ca713" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 17 15:50:45.124465 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:45.124431 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" event={"ID":"4c28f42c-b413-4a42-98e6-8759f7f4322b","Type":"ContainerStarted","Data":"5d46ed1c8c29204c019e2387f0eb583dec97bb3e7fb812326a1731324cb7a86b"} Sep 17 15:50:45.124874 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:45.124481 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" event={"ID":"4c28f42c-b413-4a42-98e6-8759f7f4322b","Type":"ContainerStarted","Data":"9f20ac032c258b473d9f8dbaab06e62128475c184cdf6fb4d96aa047f71cb1f9"} Sep 17 15:50:45.124874 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:45.124520 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" Sep 17 15:50:45.149101 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:45.149058 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" podStartSLOduration=2.149042323 podStartE2EDuration="2.149042323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:45.147374581 +0000 UTC m=+1232.143611328" watchObservedRunningTime="2026-09-17 15:50:45.149042323 +0000 UTC m=+1232.145279069" Sep 17 15:50:45.649710 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:45.649678 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:50:45.650199 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:45.650170 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:50:45.650815 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:45.650797 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 15:50:54.317772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:54.317696 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:50:56.137086 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:50:56.137057 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" Sep 17 15:51:05.255695 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:05.255659 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:06.179215 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:06.179130 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:06.506997 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:06.506889 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:06.698134 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:06.698103 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:07.104783 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:07.104737 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:07.980763 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:07.980732 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:08.902194 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:08.902165 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:09.631617 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:09.631581 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:10.382388 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:10.382352 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:11.231342 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:11.231311 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:13.586679 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:13.586651 2581 scope.go:117] "RemoveContainer" containerID="8f0f1c74bf841ca73780d7220016e231e1243cc9c00967ade7ca61f78680b357" Sep 17 15:51:19.890399 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:19.890365 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:19.987153 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:19.987127 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:25.051133 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:25.051098 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:43.987561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:43.987528 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws"] Sep 17 15:51:43.991100 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:43.991082 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:43.994114 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:43.994092 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d63c0\"/\"kube-root-ca.crt\"" Sep 17 15:51:43.994968 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:43.994949 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d63c0\"/\"openshift-service-ca.crt\"" Sep 17 15:51:43.995045 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:43.994953 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d63c0\"/\"default-dockercfg-ng4jw\"" Sep 17 15:51:43.995097 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:43.994988 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d63c0\"/\"auth-test-model-1d63c0-kserve-self-signed-certs\"" Sep 17 15:51:44.007529 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.007504 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws"] Sep 17 15:51:44.043440 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.043412 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:44.129476 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.129450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdhxh\" (UniqueName: \"kubernetes.io/projected/c6920871-10a8-4c0c-b91a-52234cda4e5c-kube-api-access-fdhxh\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.129611 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.129526 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-home\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.129611 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.129550 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-dshm\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.129611 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.129569 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6920871-10a8-4c0c-b91a-52234cda4e5c-tls-certs\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.129777 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.129622 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-tmp-dir\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.129777 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.129654 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-model-cache\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.230364 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.230330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-home\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.230557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.230369 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-dshm\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.230557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.230394 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6920871-10a8-4c0c-b91a-52234cda4e5c-tls-certs\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.230557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.230422 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-tmp-dir\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.230557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.230440 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-model-cache\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.230557 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.230473 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdhxh\" (UniqueName: \"kubernetes.io/projected/c6920871-10a8-4c0c-b91a-52234cda4e5c-kube-api-access-fdhxh\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.230857 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.230780 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-home\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.230857 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.230842 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-tmp-dir\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.230967 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.230884 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-model-cache\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.232681 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.232656 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-dshm\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.232845 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.232827 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6920871-10a8-4c0c-b91a-52234cda4e5c-tls-certs\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.239463 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.239412 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdhxh\" (UniqueName: \"kubernetes.io/projected/c6920871-10a8-4c0c-b91a-52234cda4e5c-kube-api-access-fdhxh\") pod \"auth-test-model-1d63c0-kserve-5cb6478795-ntcws\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.281755 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.281728 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:44.300758 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.300735 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:44.574997 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:44.574970 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws"] Sep 17 15:51:44.577279 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:51:44.577249 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6920871_10a8_4c0c_b91a_52234cda4e5c.slice/crio-794fd4416ee9355ce179407710a05d1a8b651943bbc61fd7809c452ca0ea41a3 WatchSource:0}: Error finding container 794fd4416ee9355ce179407710a05d1a8b651943bbc61fd7809c452ca0ea41a3: Status 404 returned error can't find the container with id 794fd4416ee9355ce179407710a05d1a8b651943bbc61fd7809c452ca0ea41a3 Sep 17 15:51:45.467022 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:45.466982 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" event={"ID":"c6920871-10a8-4c0c-b91a-52234cda4e5c","Type":"ContainerStarted","Data":"2266e80b7c6f62f2de597692665b796ba9b1bb98b0e65fb61928a0136ab765e5"} Sep 17 15:51:45.467022 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:45.467026 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" event={"ID":"c6920871-10a8-4c0c-b91a-52234cda4e5c","Type":"ContainerStarted","Data":"794fd4416ee9355ce179407710a05d1a8b651943bbc61fd7809c452ca0ea41a3"} Sep 17 15:51:45.491752 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:45.491696 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" podStartSLOduration=2.491681278 podStartE2EDuration="2.491681278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:51:45.489062634 +0000 UTC m=+1292.485299391" watchObservedRunningTime="2026-09-17 15:51:45.491681278 +0000 UTC m=+1292.487918025" Sep 17 15:51:47.411512 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:47.411478 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:51:54.301633 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:54.301591 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:54.301633 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:54.301640 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:54.314177 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:54.314146 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:51:54.528768 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:51:54.528743 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:52:04.945379 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:04.945342 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:05.662315 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:05.662283 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:06.493407 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:06.493375 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:07.267380 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:07.267342 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:07.998498 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:07.998465 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:08.968115 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:08.968083 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:09.678026 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:09.677999 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:10.537295 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:10.537264 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:11.249844 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:11.249813 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:20.488171 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.488140 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk"] Sep 17 15:52:20.488671 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.488332 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" podUID="ff1e517f-cc2e-42a2-a157-31ee8a0aeeda" containerName="maas-api" containerID="cri-o://f1e9e0f94cac800c3befe7f2fd58e8a10e38ada784bb3977584408604e4c541b" gracePeriod=30 Sep 17 15:52:20.516347 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.516321 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:20.626286 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.626250 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:20.668792 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.668764 2581 generic.go:356] "Generic (PLEG): container finished" podID="ff1e517f-cc2e-42a2-a157-31ee8a0aeeda" containerID="f1e9e0f94cac800c3befe7f2fd58e8a10e38ada784bb3977584408604e4c541b" exitCode=0 Sep 17 15:52:20.668939 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.668822 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" event={"ID":"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda","Type":"ContainerDied","Data":"f1e9e0f94cac800c3befe7f2fd58e8a10e38ada784bb3977584408604e4c541b"} Sep 17 15:52:20.740478 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.740419 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:52:20.796692 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.796653 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:20.873486 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.873460 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:20.883730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.883690 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlzxr\" (UniqueName: \"kubernetes.io/projected/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-kube-api-access-zlzxr\") pod \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " Sep 17 15:52:20.884061 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.884014 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-metrics-certs\") pod \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " Sep 17 15:52:20.884191 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.884073 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-maas-api-tls\") pod \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " Sep 17 15:52:20.884191 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.884114 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-openshift-service-ca\") pod \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\" (UID: \"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda\") " Sep 17 15:52:20.885104 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.885069 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff1e517f-cc2e-42a2-a157-31ee8a0aeeda" (UID: "ff1e517f-cc2e-42a2-a157-31ee8a0aeeda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:52:20.886612 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.886577 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-kube-api-access-zlzxr" (OuterVolumeSpecName: "kube-api-access-zlzxr") pod "ff1e517f-cc2e-42a2-a157-31ee8a0aeeda" (UID: "ff1e517f-cc2e-42a2-a157-31ee8a0aeeda"). InnerVolumeSpecName "kube-api-access-zlzxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:52:20.887036 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.887000 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff1e517f-cc2e-42a2-a157-31ee8a0aeeda" (UID: "ff1e517f-cc2e-42a2-a157-31ee8a0aeeda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:52:20.887380 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.887358 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ff1e517f-cc2e-42a2-a157-31ee8a0aeeda" (UID: "ff1e517f-cc2e-42a2-a157-31ee8a0aeeda"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:52:20.985903 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.985870 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlzxr\" (UniqueName: \"kubernetes.io/projected/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-kube-api-access-zlzxr\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:52:20.985903 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.985896 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-metrics-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:52:20.985903 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.985906 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-maas-api-tls\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:52:20.985903 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:20.985927 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda-openshift-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:52:21.176521 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:21.176486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:21.675606 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:21.675567 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" event={"ID":"ff1e517f-cc2e-42a2-a157-31ee8a0aeeda","Type":"ContainerDied","Data":"e6aafbff1445fcf8113f611ff5f2563f82d0d6165809df9a99345b1e6ee3fb41"} Sep 17 15:52:21.676037 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:21.675609 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk" Sep 17 15:52:21.676037 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:21.675635 2581 scope.go:117] "RemoveContainer" containerID="f1e9e0f94cac800c3befe7f2fd58e8a10e38ada784bb3977584408604e4c541b" Sep 17 15:52:21.706059 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:21.706008 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk"] Sep 17 15:52:21.716932 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:21.716897 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99719fde-57dfc9856c-pqbsk"] Sep 17 15:52:22.303351 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:22.303323 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:23.111196 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:23.111163 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:23.462301 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:23.462220 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff1e517f-cc2e-42a2-a157-31ee8a0aeeda" path="/var/lib/kubelet/pods/ff1e517f-cc2e-42a2-a157-31ee8a0aeeda/volumes" Sep 17 15:52:24.330714 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:24.330679 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:24.574405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:24.574376 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:25.131213 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:25.131182 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:26.147335 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:26.147302 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:27.268387 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:27.268348 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:29.621964 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:29.621930 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:44.514577 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:44.514492 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:52:49.657154 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:52:49.657113 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:00.407143 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:00.407112 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:01.713153 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:01.713122 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:06.438537 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:06.438501 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:08.970461 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:08.970429 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:09.655879 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:09.655839 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:16.274215 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:16.274180 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:35.033566 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:35.033525 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:35.154139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:35.154107 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:35.321763 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:35.321672 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89"] Sep 17 15:53:35.321971 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:35.321935 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" podUID="452e087d-09a4-467c-ae12-a1b110042ab2" containerName="istio-proxy" containerID="cri-o://4b7f0fa1a43c7a2db90a9da921b33df9d38b5fdd7b3fa7f4b0491fec1f5fdb80" gracePeriod=30 Sep 17 15:53:35.371286 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:35.371252 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:36.836351 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:36.836320 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:36.996133 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:36.996097 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:37.757930 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:37.757882 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:37.942067 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:37.942039 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:39.528600 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:39.528563 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:40.570992 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.570968 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:53:40.640835 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.640731 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxhw7\" (UniqueName: \"kubernetes.io/projected/452e087d-09a4-467c-ae12-a1b110042ab2-kube-api-access-jxhw7\") pod \"452e087d-09a4-467c-ae12-a1b110042ab2\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " Sep 17 15:53:40.640835 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.640826 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452e087d-09a4-467c-ae12-a1b110042ab2-istiod-ca-cert\") pod \"452e087d-09a4-467c-ae12-a1b110042ab2\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " Sep 17 15:53:40.641103 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.640908 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452e087d-09a4-467c-ae12-a1b110042ab2-istio-podinfo\") pod \"452e087d-09a4-467c-ae12-a1b110042ab2\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " Sep 17 15:53:40.641103 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.640974 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-workload-certs\") pod \"452e087d-09a4-467c-ae12-a1b110042ab2\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " Sep 17 15:53:40.641103 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.640997 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-istio-data\") pod \"452e087d-09a4-467c-ae12-a1b110042ab2\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " Sep 17 15:53:40.641103 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641021 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-credential-socket\") pod \"452e087d-09a4-467c-ae12-a1b110042ab2\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " Sep 17 15:53:40.641103 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641063 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452e087d-09a4-467c-ae12-a1b110042ab2-istio-token\") pod \"452e087d-09a4-467c-ae12-a1b110042ab2\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " Sep 17 15:53:40.641103 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641103 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-workload-socket\") pod \"452e087d-09a4-467c-ae12-a1b110042ab2\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " Sep 17 15:53:40.641412 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641147 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-istio-envoy\") pod \"452e087d-09a4-467c-ae12-a1b110042ab2\" (UID: \"452e087d-09a4-467c-ae12-a1b110042ab2\") " Sep 17 15:53:40.641412 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641253 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "452e087d-09a4-467c-ae12-a1b110042ab2" (UID: "452e087d-09a4-467c-ae12-a1b110042ab2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:53:40.641412 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641284 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452e087d-09a4-467c-ae12-a1b110042ab2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "452e087d-09a4-467c-ae12-a1b110042ab2" (UID: "452e087d-09a4-467c-ae12-a1b110042ab2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:53:40.641412 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641291 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "452e087d-09a4-467c-ae12-a1b110042ab2" (UID: "452e087d-09a4-467c-ae12-a1b110042ab2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:53:40.641412 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641284 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-istio-data" (OuterVolumeSpecName: "istio-data") pod "452e087d-09a4-467c-ae12-a1b110042ab2" (UID: "452e087d-09a4-467c-ae12-a1b110042ab2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:53:40.641663 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641449 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "452e087d-09a4-467c-ae12-a1b110042ab2" (UID: "452e087d-09a4-467c-ae12-a1b110042ab2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:53:40.641663 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641588 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-workload-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:53:40.641663 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641601 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-istio-data\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:53:40.641663 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641610 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-credential-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:53:40.641663 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641628 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-workload-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:53:40.641663 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.641645 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452e087d-09a4-467c-ae12-a1b110042ab2-istiod-ca-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:53:40.643312 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.643290 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "452e087d-09a4-467c-ae12-a1b110042ab2" (UID: "452e087d-09a4-467c-ae12-a1b110042ab2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:53:40.643420 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.643329 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452e087d-09a4-467c-ae12-a1b110042ab2-istio-token" (OuterVolumeSpecName: "istio-token") pod "452e087d-09a4-467c-ae12-a1b110042ab2" (UID: "452e087d-09a4-467c-ae12-a1b110042ab2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:53:40.643482 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.643422 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/452e087d-09a4-467c-ae12-a1b110042ab2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "452e087d-09a4-467c-ae12-a1b110042ab2" (UID: "452e087d-09a4-467c-ae12-a1b110042ab2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 15:53:40.643482 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.643427 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452e087d-09a4-467c-ae12-a1b110042ab2-kube-api-access-jxhw7" (OuterVolumeSpecName: "kube-api-access-jxhw7") pod "452e087d-09a4-467c-ae12-a1b110042ab2" (UID: "452e087d-09a4-467c-ae12-a1b110042ab2"). InnerVolumeSpecName "kube-api-access-jxhw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:53:40.742550 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.742516 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452e087d-09a4-467c-ae12-a1b110042ab2-istio-podinfo\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:53:40.742550 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.742544 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452e087d-09a4-467c-ae12-a1b110042ab2-istio-token\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:53:40.742550 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.742555 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452e087d-09a4-467c-ae12-a1b110042ab2-istio-envoy\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:53:40.742786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:40.742563 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxhw7\" (UniqueName: \"kubernetes.io/projected/452e087d-09a4-467c-ae12-a1b110042ab2-kube-api-access-jxhw7\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:53:41.123375 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:41.123341 2581 generic.go:356] "Generic (PLEG): container finished" podID="452e087d-09a4-467c-ae12-a1b110042ab2" containerID="4b7f0fa1a43c7a2db90a9da921b33df9d38b5fdd7b3fa7f4b0491fec1f5fdb80" exitCode=0 Sep 17 15:53:41.123566 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:41.123387 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" Sep 17 15:53:41.123566 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:41.123421 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" event={"ID":"452e087d-09a4-467c-ae12-a1b110042ab2","Type":"ContainerDied","Data":"4b7f0fa1a43c7a2db90a9da921b33df9d38b5fdd7b3fa7f4b0491fec1f5fdb80"} Sep 17 15:53:41.123566 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:41.123474 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89" event={"ID":"452e087d-09a4-467c-ae12-a1b110042ab2","Type":"ContainerDied","Data":"f1ad19921feb49dd9ad8406d8ec489e768b2dc916065e24347a72f59bd5fddc4"} Sep 17 15:53:41.123566 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:41.123497 2581 scope.go:117] "RemoveContainer" containerID="4b7f0fa1a43c7a2db90a9da921b33df9d38b5fdd7b3fa7f4b0491fec1f5fdb80" Sep 17 15:53:41.133398 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:41.133379 2581 scope.go:117] "RemoveContainer" containerID="bebb145ca75fbdc5904f1c10ea4696439efb54135cba250783d895911fd3dcf0" Sep 17 15:53:41.141754 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:41.141691 2581 scope.go:117] "RemoveContainer" containerID="4b7f0fa1a43c7a2db90a9da921b33df9d38b5fdd7b3fa7f4b0491fec1f5fdb80" Sep 17 15:53:41.142484 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:53:41.142437 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7f0fa1a43c7a2db90a9da921b33df9d38b5fdd7b3fa7f4b0491fec1f5fdb80\": container with ID starting with 4b7f0fa1a43c7a2db90a9da921b33df9d38b5fdd7b3fa7f4b0491fec1f5fdb80 not found: ID does not exist" containerID="4b7f0fa1a43c7a2db90a9da921b33df9d38b5fdd7b3fa7f4b0491fec1f5fdb80" Sep 17 15:53:41.142596 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:41.142478 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7f0fa1a43c7a2db90a9da921b33df9d38b5fdd7b3fa7f4b0491fec1f5fdb80"} err="failed to get container status \"4b7f0fa1a43c7a2db90a9da921b33df9d38b5fdd7b3fa7f4b0491fec1f5fdb80\": rpc error: code = NotFound desc = could not find container \"4b7f0fa1a43c7a2db90a9da921b33df9d38b5fdd7b3fa7f4b0491fec1f5fdb80\": container with ID starting with 4b7f0fa1a43c7a2db90a9da921b33df9d38b5fdd7b3fa7f4b0491fec1f5fdb80 not found: ID does not exist" Sep 17 15:53:41.142596 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:41.142537 2581 scope.go:117] "RemoveContainer" containerID="bebb145ca75fbdc5904f1c10ea4696439efb54135cba250783d895911fd3dcf0" Sep 17 15:53:41.142976 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:53:41.142948 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebb145ca75fbdc5904f1c10ea4696439efb54135cba250783d895911fd3dcf0\": container with ID starting with bebb145ca75fbdc5904f1c10ea4696439efb54135cba250783d895911fd3dcf0 not found: ID does not exist" containerID="bebb145ca75fbdc5904f1c10ea4696439efb54135cba250783d895911fd3dcf0" Sep 17 15:53:41.143090 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:41.142983 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebb145ca75fbdc5904f1c10ea4696439efb54135cba250783d895911fd3dcf0"} err="failed to get container status \"bebb145ca75fbdc5904f1c10ea4696439efb54135cba250783d895911fd3dcf0\": rpc error: code = NotFound desc = could not find container \"bebb145ca75fbdc5904f1c10ea4696439efb54135cba250783d895911fd3dcf0\": container with ID starting with bebb145ca75fbdc5904f1c10ea4696439efb54135cba250783d895911fd3dcf0 not found: ID does not exist" Sep 17 15:53:41.165676 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:41.165644 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89"] Sep 17 15:53:41.172987 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:41.172963 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-99719fde-openshift-default-595b55748-pjw89"] Sep 17 15:53:41.462712 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:41.462624 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="452e087d-09a4-467c-ae12-a1b110042ab2" path="/var/lib/kubelet/pods/452e087d-09a4-467c-ae12-a1b110042ab2/volumes" Sep 17 15:53:46.896177 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:46.896146 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:46.996990 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:46.996957 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:47.715435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:47.715389 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:54.674936 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:54.674887 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:54.787252 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:54.787215 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:53:54.954763 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:53:54.954668 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:05.483488 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:05.483445 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:05.709256 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:05.709216 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:14.797654 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:14.797615 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:19.812087 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:19.812042 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:20.290224 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:20.290192 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:21.189320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:21.189279 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:22.159086 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:22.159056 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:22.862514 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:22.862475 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:23.839172 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:23.839142 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:24.713706 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:24.713675 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:24.847052 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:24.847016 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh"] Sep 17 15:54:24.920148 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:24.920079 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:25.589471 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:25.589435 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:26.511580 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:26.511514 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:26.693085 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:26.693055 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:33.081268 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.081233 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:33.182249 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.182213 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:33.434301 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.434186 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:33.745783 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.745703 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw"] Sep 17 15:54:33.746118 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.746105 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452e087d-09a4-467c-ae12-a1b110042ab2" containerName="istio-proxy" Sep 17 15:54:33.746176 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.746119 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="452e087d-09a4-467c-ae12-a1b110042ab2" containerName="istio-proxy" Sep 17 15:54:33.746176 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.746131 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1e517f-cc2e-42a2-a157-31ee8a0aeeda" containerName="maas-api" Sep 17 15:54:33.746176 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.746137 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1e517f-cc2e-42a2-a157-31ee8a0aeeda" containerName="maas-api" Sep 17 15:54:33.746269 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.746202 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff1e517f-cc2e-42a2-a157-31ee8a0aeeda" containerName="maas-api" Sep 17 15:54:33.746269 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.746209 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="452e087d-09a4-467c-ae12-a1b110042ab2" containerName="istio-proxy" Sep 17 15:54:33.746269 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.746216 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="452e087d-09a4-467c-ae12-a1b110042ab2" containerName="istio-proxy" Sep 17 15:54:33.749522 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.749502 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.753237 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.753221 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c686d0-openshift-default-dockercfg-4lvvh\"" Sep 17 15:54:33.769146 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.769123 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw"] Sep 17 15:54:33.777156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.777124 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:33.825271 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.825235 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-workload-socket\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.825477 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.825281 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-podinfo\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.825477 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.825436 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-data\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.825603 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.825474 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-credential-socket\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.825603 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.825510 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws9g4\" (UniqueName: \"kubernetes.io/projected/82906d2d-5c60-4e02-829f-d010c89a70f1-kube-api-access-ws9g4\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.825603 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.825539 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-token\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.825603 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.825568 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-envoy\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.825782 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.825611 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-workload-certs\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.825782 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.825646 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82906d2d-5c60-4e02-829f-d010c89a70f1-istiod-ca-cert\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.908794 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.908748 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:33.927186 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.927150 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-podinfo\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.927544 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.927511 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-data\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.927658 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.927557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-credential-socket\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.927658 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.927589 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws9g4\" (UniqueName: \"kubernetes.io/projected/82906d2d-5c60-4e02-829f-d010c89a70f1-kube-api-access-ws9g4\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.927658 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.927618 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-token\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.927658 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.927647 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-envoy\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.927893 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.927693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-workload-certs\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.927893 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.927730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82906d2d-5c60-4e02-829f-d010c89a70f1-istiod-ca-cert\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.927893 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.927765 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-workload-socket\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.928095 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.928057 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-credential-socket\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.928156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.928139 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-workload-socket\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.928457 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.928426 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-data\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.928552 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.928486 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-workload-certs\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.928787 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.928750 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82906d2d-5c60-4e02-829f-d010c89a70f1-istiod-ca-cert\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.930366 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.930344 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-podinfo\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.930943 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.930898 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-envoy\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.941075 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.941047 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws9g4\" (UniqueName: \"kubernetes.io/projected/82906d2d-5c60-4e02-829f-d010c89a70f1-kube-api-access-ws9g4\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:33.942968 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:33.942943 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-token\") pod \"e2e-api-a-c686d0-openshift-default-79744f8746-j49hw\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:34.060178 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:34.060148 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:34.136121 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:34.136092 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:34.338870 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:34.338840 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw"] Sep 17 15:54:34.340971 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:54:34.340944 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82906d2d_5c60_4e02_829f_d010c89a70f1.slice/crio-a1c2e7993ed5e44802626a66affb7853b2526fe7f28c72b411dc18dccbcfaf96 WatchSource:0}: Error finding container a1c2e7993ed5e44802626a66affb7853b2526fe7f28c72b411dc18dccbcfaf96: Status 404 returned error can't find the container with id a1c2e7993ed5e44802626a66affb7853b2526fe7f28c72b411dc18dccbcfaf96 Sep 17 15:54:34.343236 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:34.343206 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:54:34.343303 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:34.343270 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:54:34.343351 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:34.343314 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:54:34.425228 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:34.425192 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" event={"ID":"82906d2d-5c60-4e02-829f-d010c89a70f1","Type":"ContainerStarted","Data":"fc9d5068c27ab4d61bc489df8dfe0cdc53c99a4ca97f66d90ed22f0473093414"} Sep 17 15:54:34.425390 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:34.425246 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" event={"ID":"82906d2d-5c60-4e02-829f-d010c89a70f1","Type":"ContainerStarted","Data":"a1c2e7993ed5e44802626a66affb7853b2526fe7f28c72b411dc18dccbcfaf96"} Sep 17 15:54:34.461813 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:34.461763 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" podStartSLOduration=1.461740891 podStartE2EDuration="1.461740891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:34.460082688 +0000 UTC m=+1461.456319475" watchObservedRunningTime="2026-09-17 15:54:34.461740891 +0000 UTC m=+1461.457977639" Sep 17 15:54:34.651243 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:34.651168 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:35.060738 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:35.060706 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:35.065843 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:35.065815 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:35.431520 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:35.431440 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:35.432584 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:35.432555 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:54:35.625315 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:35.625288 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:36.380861 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:36.380822 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:36.670326 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:36.670232 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:36.783822 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:36.783791 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:37.765273 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:37.765234 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:38.453588 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.453558 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr"] Sep 17 15:54:38.454230 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.453763 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" podUID="cf6df8da-ce1f-45b9-b416-a82cda25f5f2" containerName="maas-api" containerID="cri-o://8176556b1872c27434e658c0b9d09bbee17a1ed581cbb970f96edbbc5cb073ee" gracePeriod=30 Sep 17 15:54:38.705695 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.705620 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:54:38.778436 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.778401 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-metrics-certs\") pod \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " Sep 17 15:54:38.778860 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.778454 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-openshift-service-ca\") pod \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " Sep 17 15:54:38.778860 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.778495 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-maas-api-tls\") pod \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " Sep 17 15:54:38.778860 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.778555 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl6qw\" (UniqueName: \"kubernetes.io/projected/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-kube-api-access-pl6qw\") pod \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\" (UID: \"cf6df8da-ce1f-45b9-b416-a82cda25f5f2\") " Sep 17 15:54:38.779042 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.778855 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cf6df8da-ce1f-45b9-b416-a82cda25f5f2" (UID: "cf6df8da-ce1f-45b9-b416-a82cda25f5f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:54:38.780524 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.780491 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cf6df8da-ce1f-45b9-b416-a82cda25f5f2" (UID: "cf6df8da-ce1f-45b9-b416-a82cda25f5f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:54:38.780645 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.780621 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cf6df8da-ce1f-45b9-b416-a82cda25f5f2" (UID: "cf6df8da-ce1f-45b9-b416-a82cda25f5f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:54:38.780705 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.780654 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-kube-api-access-pl6qw" (OuterVolumeSpecName: "kube-api-access-pl6qw") pod "cf6df8da-ce1f-45b9-b416-a82cda25f5f2" (UID: "cf6df8da-ce1f-45b9-b416-a82cda25f5f2"). InnerVolumeSpecName "kube-api-access-pl6qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:54:38.880230 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.880196 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-metrics-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:38.880230 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.880225 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-openshift-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:38.880230 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.880236 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-maas-api-tls\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:38.880481 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:38.880245 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl6qw\" (UniqueName: \"kubernetes.io/projected/cf6df8da-ce1f-45b9-b416-a82cda25f5f2-kube-api-access-pl6qw\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:39.143339 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.143301 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh"] Sep 17 15:54:39.207772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.207731 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws"] Sep 17 15:54:39.456051 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.455909 2581 generic.go:356] "Generic (PLEG): container finished" podID="cf6df8da-ce1f-45b9-b416-a82cda25f5f2" containerID="8176556b1872c27434e658c0b9d09bbee17a1ed581cbb970f96edbbc5cb073ee" exitCode=0 Sep 17 15:54:39.456051 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.455989 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" Sep 17 15:54:39.462545 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.462509 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" event={"ID":"cf6df8da-ce1f-45b9-b416-a82cda25f5f2","Type":"ContainerDied","Data":"8176556b1872c27434e658c0b9d09bbee17a1ed581cbb970f96edbbc5cb073ee"} Sep 17 15:54:39.462712 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.462555 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr" event={"ID":"cf6df8da-ce1f-45b9-b416-a82cda25f5f2","Type":"ContainerDied","Data":"d911de2bb17d21c964a0e1d76757c64f77b59ba32c7b9225662d61caf585b91e"} Sep 17 15:54:39.462712 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.462587 2581 scope.go:117] "RemoveContainer" containerID="8176556b1872c27434e658c0b9d09bbee17a1ed581cbb970f96edbbc5cb073ee" Sep 17 15:54:39.472575 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.472554 2581 scope.go:117] "RemoveContainer" containerID="8176556b1872c27434e658c0b9d09bbee17a1ed581cbb970f96edbbc5cb073ee" Sep 17 15:54:39.472849 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:54:39.472827 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8176556b1872c27434e658c0b9d09bbee17a1ed581cbb970f96edbbc5cb073ee\": container with ID starting with 8176556b1872c27434e658c0b9d09bbee17a1ed581cbb970f96edbbc5cb073ee not found: ID does not exist" containerID="8176556b1872c27434e658c0b9d09bbee17a1ed581cbb970f96edbbc5cb073ee" Sep 17 15:54:39.472901 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.472858 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8176556b1872c27434e658c0b9d09bbee17a1ed581cbb970f96edbbc5cb073ee"} err="failed to get container status \"8176556b1872c27434e658c0b9d09bbee17a1ed581cbb970f96edbbc5cb073ee\": rpc error: code = NotFound desc = could not find container \"8176556b1872c27434e658c0b9d09bbee17a1ed581cbb970f96edbbc5cb073ee\": container with ID starting with 8176556b1872c27434e658c0b9d09bbee17a1ed581cbb970f96edbbc5cb073ee not found: ID does not exist" Sep 17 15:54:39.487076 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.487050 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr"] Sep 17 15:54:39.490875 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.490854 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c86e9a-7bd47c6588-nnpmr"] Sep 17 15:54:39.733881 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.733793 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:39.866448 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.866404 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" podUID="799214bd-9f3f-46a1-b3c5-b2a2b54a4d51" containerName="main" containerID="cri-o://e4809dacf664772296cde72dd45162ab7296e755e1c0b4e5f2ec8c277e7721e5" gracePeriod=45 Sep 17 15:54:39.878494 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:39.878470 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:40.122000 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.121967 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:54:40.194712 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.194679 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77s9s\" (UniqueName: \"kubernetes.io/projected/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-kube-api-access-77s9s\") pod \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " Sep 17 15:54:40.194872 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.194755 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-model-cache\") pod \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " Sep 17 15:54:40.194872 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.194833 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-dshm\") pod \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " Sep 17 15:54:40.195009 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.194890 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-tmp-dir\") pod \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " Sep 17 15:54:40.195009 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.194959 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-home\") pod \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " Sep 17 15:54:40.195009 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.195002 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-tls-certs\") pod \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\" (UID: \"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51\") " Sep 17 15:54:40.195167 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.195044 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-model-cache" (OuterVolumeSpecName: "model-cache") pod "799214bd-9f3f-46a1-b3c5-b2a2b54a4d51" (UID: "799214bd-9f3f-46a1-b3c5-b2a2b54a4d51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:54:40.195226 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.195177 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "799214bd-9f3f-46a1-b3c5-b2a2b54a4d51" (UID: "799214bd-9f3f-46a1-b3c5-b2a2b54a4d51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:54:40.195286 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.195271 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-home" (OuterVolumeSpecName: "home") pod "799214bd-9f3f-46a1-b3c5-b2a2b54a4d51" (UID: "799214bd-9f3f-46a1-b3c5-b2a2b54a4d51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:54:40.195345 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.195323 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-model-cache\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:40.195345 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.195339 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-tmp-dir\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:40.197309 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.197281 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "799214bd-9f3f-46a1-b3c5-b2a2b54a4d51" (UID: "799214bd-9f3f-46a1-b3c5-b2a2b54a4d51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:54:40.197440 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.197027 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-kube-api-access-77s9s" (OuterVolumeSpecName: "kube-api-access-77s9s") pod "799214bd-9f3f-46a1-b3c5-b2a2b54a4d51" (UID: "799214bd-9f3f-46a1-b3c5-b2a2b54a4d51"). InnerVolumeSpecName "kube-api-access-77s9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:54:40.197664 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.197640 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-dshm" (OuterVolumeSpecName: "dshm") pod "799214bd-9f3f-46a1-b3c5-b2a2b54a4d51" (UID: "799214bd-9f3f-46a1-b3c5-b2a2b54a4d51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:54:40.290661 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.290575 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:40.296247 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.296225 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-home\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:40.296326 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.296250 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-tls-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:40.296326 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.296261 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77s9s\" (UniqueName: \"kubernetes.io/projected/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-kube-api-access-77s9s\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:40.296326 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.296269 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51-dshm\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:40.464274 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.464243 2581 generic.go:356] "Generic (PLEG): container finished" podID="799214bd-9f3f-46a1-b3c5-b2a2b54a4d51" containerID="e4809dacf664772296cde72dd45162ab7296e755e1c0b4e5f2ec8c277e7721e5" exitCode=0 Sep 17 15:54:40.464459 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.464284 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" Sep 17 15:54:40.464459 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.464295 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" event={"ID":"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51","Type":"ContainerDied","Data":"e4809dacf664772296cde72dd45162ab7296e755e1c0b4e5f2ec8c277e7721e5"} Sep 17 15:54:40.464459 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.464325 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh" event={"ID":"799214bd-9f3f-46a1-b3c5-b2a2b54a4d51","Type":"ContainerDied","Data":"42e8102bb4b6ba3c59c988df392dd0c003f682940a669aa14c1294d8dfaf85f0"} Sep 17 15:54:40.464459 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.464346 2581 scope.go:117] "RemoveContainer" containerID="e4809dacf664772296cde72dd45162ab7296e755e1c0b4e5f2ec8c277e7721e5" Sep 17 15:54:40.474258 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.474238 2581 scope.go:117] "RemoveContainer" containerID="e4809dacf664772296cde72dd45162ab7296e755e1c0b4e5f2ec8c277e7721e5" Sep 17 15:54:40.474501 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:54:40.474483 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4809dacf664772296cde72dd45162ab7296e755e1c0b4e5f2ec8c277e7721e5\": container with ID starting with e4809dacf664772296cde72dd45162ab7296e755e1c0b4e5f2ec8c277e7721e5 not found: ID does not exist" containerID="e4809dacf664772296cde72dd45162ab7296e755e1c0b4e5f2ec8c277e7721e5" Sep 17 15:54:40.474563 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.474511 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4809dacf664772296cde72dd45162ab7296e755e1c0b4e5f2ec8c277e7721e5"} err="failed to get container status \"e4809dacf664772296cde72dd45162ab7296e755e1c0b4e5f2ec8c277e7721e5\": rpc error: code = NotFound desc = could not find container \"e4809dacf664772296cde72dd45162ab7296e755e1c0b4e5f2ec8c277e7721e5\": container with ID starting with e4809dacf664772296cde72dd45162ab7296e755e1c0b4e5f2ec8c277e7721e5 not found: ID does not exist" Sep 17 15:54:40.494442 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.494414 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh"] Sep 17 15:54:40.499287 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:40.499264 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/simulator-w4-c86e9a-kserve-66fc66865c-txmlh"] Sep 17 15:54:41.123244 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:41.123208 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:41.462354 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:41.462277 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="799214bd-9f3f-46a1-b3c5-b2a2b54a4d51" path="/var/lib/kubelet/pods/799214bd-9f3f-46a1-b3c5-b2a2b54a4d51/volumes" Sep 17 15:54:41.462708 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:41.462695 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf6df8da-ce1f-45b9-b416-a82cda25f5f2" path="/var/lib/kubelet/pods/cf6df8da-ce1f-45b9-b416-a82cda25f5f2/volumes" Sep 17 15:54:41.951974 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:41.951941 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:42.052550 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.052514 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:42.372659 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.372628 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:42.573029 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.572992 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:42.720079 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.719999 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:42.759672 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.759639 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5"] Sep 17 15:54:42.760032 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.760019 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452e087d-09a4-467c-ae12-a1b110042ab2" containerName="istio-proxy" Sep 17 15:54:42.760091 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.760033 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="452e087d-09a4-467c-ae12-a1b110042ab2" containerName="istio-proxy" Sep 17 15:54:42.760091 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.760042 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf6df8da-ce1f-45b9-b416-a82cda25f5f2" containerName="maas-api" Sep 17 15:54:42.760091 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.760048 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf6df8da-ce1f-45b9-b416-a82cda25f5f2" containerName="maas-api" Sep 17 15:54:42.760091 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.760059 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="799214bd-9f3f-46a1-b3c5-b2a2b54a4d51" containerName="main" Sep 17 15:54:42.760091 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.760064 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="799214bd-9f3f-46a1-b3c5-b2a2b54a4d51" containerName="main" Sep 17 15:54:42.760269 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.760131 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="799214bd-9f3f-46a1-b3c5-b2a2b54a4d51" containerName="main" Sep 17 15:54:42.760269 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.760139 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf6df8da-ce1f-45b9-b416-a82cda25f5f2" containerName="maas-api" Sep 17 15:54:42.764327 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.764312 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" Sep 17 15:54:42.766774 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.766752 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c686d0-dockercfg-9zg4n\"" Sep 17 15:54:42.767220 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.767206 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c686d0\"" Sep 17 15:54:42.796151 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.796118 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5"] Sep 17 15:54:42.821662 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.821635 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e088820e-fecb-45a1-8883-c58802b693f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5\" (UID: \"e088820e-fecb-45a1-8883-c58802b693f0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" Sep 17 15:54:42.821777 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.821752 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j8p8\" (UniqueName: \"kubernetes.io/projected/e088820e-fecb-45a1-8883-c58802b693f0-kube-api-access-7j8p8\") pod \"payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5\" (UID: \"e088820e-fecb-45a1-8883-c58802b693f0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" Sep 17 15:54:42.848088 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.848062 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx"] Sep 17 15:54:42.851599 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.851582 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" Sep 17 15:54:42.868783 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.868747 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx"] Sep 17 15:54:42.922697 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.922663 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j8p8\" (UniqueName: \"kubernetes.io/projected/e088820e-fecb-45a1-8883-c58802b693f0-kube-api-access-7j8p8\") pod \"payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5\" (UID: \"e088820e-fecb-45a1-8883-c58802b693f0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" Sep 17 15:54:42.922697 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.922701 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km266\" (UniqueName: \"kubernetes.io/projected/cca909d6-fef7-4bdd-b472-e83fbe704f62-kube-api-access-km266\") pod \"payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx\" (UID: \"cca909d6-fef7-4bdd-b472-e83fbe704f62\") " pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" Sep 17 15:54:42.922938 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.922757 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca909d6-fef7-4bdd-b472-e83fbe704f62-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx\" (UID: \"cca909d6-fef7-4bdd-b472-e83fbe704f62\") " pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" Sep 17 15:54:42.922938 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.922790 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e088820e-fecb-45a1-8883-c58802b693f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5\" (UID: \"e088820e-fecb-45a1-8883-c58802b693f0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" Sep 17 15:54:42.923469 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.923449 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e088820e-fecb-45a1-8883-c58802b693f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5\" (UID: \"e088820e-fecb-45a1-8883-c58802b693f0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" Sep 17 15:54:42.932112 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:42.932084 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j8p8\" (UniqueName: \"kubernetes.io/projected/e088820e-fecb-45a1-8883-c58802b693f0-kube-api-access-7j8p8\") pod \"payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5\" (UID: \"e088820e-fecb-45a1-8883-c58802b693f0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" Sep 17 15:54:43.024017 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.023900 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km266\" (UniqueName: \"kubernetes.io/projected/cca909d6-fef7-4bdd-b472-e83fbe704f62-kube-api-access-km266\") pod \"payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx\" (UID: \"cca909d6-fef7-4bdd-b472-e83fbe704f62\") " pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" Sep 17 15:54:43.024017 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.024004 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca909d6-fef7-4bdd-b472-e83fbe704f62-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx\" (UID: \"cca909d6-fef7-4bdd-b472-e83fbe704f62\") " pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" Sep 17 15:54:43.024675 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.024657 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca909d6-fef7-4bdd-b472-e83fbe704f62-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx\" (UID: \"cca909d6-fef7-4bdd-b472-e83fbe704f62\") " pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" Sep 17 15:54:43.033934 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.033881 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km266\" (UniqueName: \"kubernetes.io/projected/cca909d6-fef7-4bdd-b472-e83fbe704f62-kube-api-access-km266\") pod \"payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx\" (UID: \"cca909d6-fef7-4bdd-b472-e83fbe704f62\") " pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" Sep 17 15:54:43.074023 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.073990 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" Sep 17 15:54:43.162834 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.162795 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" Sep 17 15:54:43.203397 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.203313 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:43.350592 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.350567 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5"] Sep 17 15:54:43.351503 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:54:43.351473 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode088820e_fecb_45a1_8883_c58802b693f0.slice/crio-bdeb4eb5d24139b1a3ef6ea3d92db08e3ffa2a5dc60374ef5ca8e2a303d3617f WatchSource:0}: Error finding container bdeb4eb5d24139b1a3ef6ea3d92db08e3ffa2a5dc60374ef5ca8e2a303d3617f: Status 404 returned error can't find the container with id bdeb4eb5d24139b1a3ef6ea3d92db08e3ffa2a5dc60374ef5ca8e2a303d3617f Sep 17 15:54:43.440787 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.440759 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx"] Sep 17 15:54:43.441645 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:54:43.441621 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca909d6_fef7_4bdd_b472_e83fbe704f62.slice/crio-669ad78c16fe0b87f851cc9487fe1c6c9f2d3cebc897bc3167dcbd1cf1e0e5a7 WatchSource:0}: Error finding container 669ad78c16fe0b87f851cc9487fe1c6c9f2d3cebc897bc3167dcbd1cf1e0e5a7: Status 404 returned error can't find the container with id 669ad78c16fe0b87f851cc9487fe1c6c9f2d3cebc897bc3167dcbd1cf1e0e5a7 Sep 17 15:54:43.487461 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.487426 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" event={"ID":"e088820e-fecb-45a1-8883-c58802b693f0","Type":"ContainerStarted","Data":"ab33c87b41dd03e55eb3c0f2f96db6fe2b14b76a06cef5234be824afa4d37a51"} Sep 17 15:54:43.487628 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.487492 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" event={"ID":"e088820e-fecb-45a1-8883-c58802b693f0","Type":"ContainerStarted","Data":"bdeb4eb5d24139b1a3ef6ea3d92db08e3ffa2a5dc60374ef5ca8e2a303d3617f"} Sep 17 15:54:43.488423 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.488352 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" Sep 17 15:54:43.490397 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.490219 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" event={"ID":"cca909d6-fef7-4bdd-b472-e83fbe704f62","Type":"ContainerStarted","Data":"669ad78c16fe0b87f851cc9487fe1c6c9f2d3cebc897bc3167dcbd1cf1e0e5a7"} Sep 17 15:54:43.508320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:43.508276 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" podStartSLOduration=1.508258726 podStartE2EDuration="1.508258726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:43.505451641 +0000 UTC m=+1470.501688381" watchObservedRunningTime="2026-09-17 15:54:43.508258726 +0000 UTC m=+1470.504495472" Sep 17 15:54:44.145257 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:44.145225 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:44.499813 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:44.499716 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" event={"ID":"cca909d6-fef7-4bdd-b472-e83fbe704f62","Type":"ContainerStarted","Data":"90c51eb62548adb69e54b6bff3381fc361c76abf11b376d3522e8d0b6996773a"} Sep 17 15:54:44.500250 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:44.499808 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" Sep 17 15:54:44.523787 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:44.523721 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" podStartSLOduration=2.5237043359999998 podStartE2EDuration="2.523704336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:44.520681843 +0000 UTC m=+1471.516918591" watchObservedRunningTime="2026-09-17 15:54:44.523704336 +0000 UTC m=+1471.519941082" Sep 17 15:54:45.137183 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:45.137153 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:45.847280 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:45.847246 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:45.949663 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:45.949623 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:46.157290 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:46.157208 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:46.774895 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:46.774862 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:50.102893 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:50.102860 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:52.669589 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:52.669554 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:52.726569 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:52.726532 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5"] Sep 17 15:54:52.726795 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:52.726769 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" podUID="e0537ab3-dc97-4bd8-adec-f6b1ccdda18e" containerName="payload-pre-processing" containerID="cri-o://13406c072373bbfc622d7b7c108ea3a63005acb6056109b80fb45b23d9a26df4" gracePeriod=30 Sep 17 15:54:52.767978 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:52.767937 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:52.869890 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:52.869855 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:52.981350 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:52.981320 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" Sep 17 15:54:52.983122 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:52.983047 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:53.124993 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.124945 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4twgk\" (UniqueName: \"kubernetes.io/projected/e0537ab3-dc97-4bd8-adec-f6b1ccdda18e-kube-api-access-4twgk\") pod \"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e\" (UID: \"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e\") " Sep 17 15:54:53.124993 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.124995 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0537ab3-dc97-4bd8-adec-f6b1ccdda18e-plugins-config-volume\") pod \"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e\" (UID: \"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e\") " Sep 17 15:54:53.125359 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.125336 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0537ab3-dc97-4bd8-adec-f6b1ccdda18e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0537ab3-dc97-4bd8-adec-f6b1ccdda18e" (UID: "e0537ab3-dc97-4bd8-adec-f6b1ccdda18e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:54:53.126953 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.126929 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0537ab3-dc97-4bd8-adec-f6b1ccdda18e-kube-api-access-4twgk" (OuterVolumeSpecName: "kube-api-access-4twgk") pod "e0537ab3-dc97-4bd8-adec-f6b1ccdda18e" (UID: "e0537ab3-dc97-4bd8-adec-f6b1ccdda18e"). InnerVolumeSpecName "kube-api-access-4twgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:54:53.226736 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.226655 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4twgk\" (UniqueName: \"kubernetes.io/projected/e0537ab3-dc97-4bd8-adec-f6b1ccdda18e-kube-api-access-4twgk\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:53.226736 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.226694 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0537ab3-dc97-4bd8-adec-f6b1ccdda18e-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:53.309847 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.309811 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:53.551137 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.551100 2581 generic.go:356] "Generic (PLEG): container finished" podID="e0537ab3-dc97-4bd8-adec-f6b1ccdda18e" containerID="13406c072373bbfc622d7b7c108ea3a63005acb6056109b80fb45b23d9a26df4" exitCode=0 Sep 17 15:54:53.551311 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.551157 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" Sep 17 15:54:53.551311 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.551180 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" event={"ID":"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e","Type":"ContainerDied","Data":"13406c072373bbfc622d7b7c108ea3a63005acb6056109b80fb45b23d9a26df4"} Sep 17 15:54:53.551311 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.551247 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5" event={"ID":"e0537ab3-dc97-4bd8-adec-f6b1ccdda18e","Type":"ContainerDied","Data":"5ab78939f7d279d04a2a47a1f8a0f31cfb674b8be6d32f4fe53944f3257af3af"} Sep 17 15:54:53.551311 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.551278 2581 scope.go:117] "RemoveContainer" containerID="13406c072373bbfc622d7b7c108ea3a63005acb6056109b80fb45b23d9a26df4" Sep 17 15:54:53.560473 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.560457 2581 scope.go:117] "RemoveContainer" containerID="13406c072373bbfc622d7b7c108ea3a63005acb6056109b80fb45b23d9a26df4" Sep 17 15:54:53.560756 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:54:53.560738 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13406c072373bbfc622d7b7c108ea3a63005acb6056109b80fb45b23d9a26df4\": container with ID starting with 13406c072373bbfc622d7b7c108ea3a63005acb6056109b80fb45b23d9a26df4 not found: ID does not exist" containerID="13406c072373bbfc622d7b7c108ea3a63005acb6056109b80fb45b23d9a26df4" Sep 17 15:54:53.560801 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.560765 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13406c072373bbfc622d7b7c108ea3a63005acb6056109b80fb45b23d9a26df4"} err="failed to get container status \"13406c072373bbfc622d7b7c108ea3a63005acb6056109b80fb45b23d9a26df4\": rpc error: code = NotFound desc = could not find container \"13406c072373bbfc622d7b7c108ea3a63005acb6056109b80fb45b23d9a26df4\": container with ID starting with 13406c072373bbfc622d7b7c108ea3a63005acb6056109b80fb45b23d9a26df4 not found: ID does not exist" Sep 17 15:54:53.567834 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.567810 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5"] Sep 17 15:54:53.573152 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.573132 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-849b71-77497df66-xs6c5"] Sep 17 15:54:53.816545 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:53.816461 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:54.162309 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.162225 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" podUID="25424ac6-a827-4f63-abb3-58a873ff27fa" containerName="main" containerID="cri-o://c25a8e17d3a8195e864baab455c283f181b5ebbdb5d574dd1df9531467f18773" gracePeriod=45 Sep 17 15:54:54.226926 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.226880 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" podUID="c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073" containerName="main" containerID="cri-o://104ac2092d0c94a68daddf7e43dfb6007ec55cca436d9f5704d812df9dbd2bba" gracePeriod=45 Sep 17 15:54:54.494577 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.494556 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:54:54.497464 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.497446 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:54:54.561784 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.561750 2581 generic.go:356] "Generic (PLEG): container finished" podID="25424ac6-a827-4f63-abb3-58a873ff27fa" containerID="c25a8e17d3a8195e864baab455c283f181b5ebbdb5d574dd1df9531467f18773" exitCode=0 Sep 17 15:54:54.561981 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.561811 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" Sep 17 15:54:54.561981 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.561834 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" event={"ID":"25424ac6-a827-4f63-abb3-58a873ff27fa","Type":"ContainerDied","Data":"c25a8e17d3a8195e864baab455c283f181b5ebbdb5d574dd1df9531467f18773"} Sep 17 15:54:54.561981 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.561876 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh" event={"ID":"25424ac6-a827-4f63-abb3-58a873ff27fa","Type":"ContainerDied","Data":"fae10e79e38d17feea26a6938c9f032a3a31dd4cf870cb5088f22cd628885b08"} Sep 17 15:54:54.561981 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.561899 2581 scope.go:117] "RemoveContainer" containerID="c25a8e17d3a8195e864baab455c283f181b5ebbdb5d574dd1df9531467f18773" Sep 17 15:54:54.563989 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.563961 2581 generic.go:356] "Generic (PLEG): container finished" podID="c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073" containerID="104ac2092d0c94a68daddf7e43dfb6007ec55cca436d9f5704d812df9dbd2bba" exitCode=0 Sep 17 15:54:54.564115 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.564068 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" Sep 17 15:54:54.564115 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.564082 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" event={"ID":"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073","Type":"ContainerDied","Data":"104ac2092d0c94a68daddf7e43dfb6007ec55cca436d9f5704d812df9dbd2bba"} Sep 17 15:54:54.564244 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.564115 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws" event={"ID":"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073","Type":"ContainerDied","Data":"8d1bb12c15200e641d575e342f4fe4d776d14a89e99e8c968cbc43796116c1f3"} Sep 17 15:54:54.571756 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.571691 2581 scope.go:117] "RemoveContainer" containerID="c25a8e17d3a8195e864baab455c283f181b5ebbdb5d574dd1df9531467f18773" Sep 17 15:54:54.572048 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:54:54.572025 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c25a8e17d3a8195e864baab455c283f181b5ebbdb5d574dd1df9531467f18773\": container with ID starting with c25a8e17d3a8195e864baab455c283f181b5ebbdb5d574dd1df9531467f18773 not found: ID does not exist" containerID="c25a8e17d3a8195e864baab455c283f181b5ebbdb5d574dd1df9531467f18773" Sep 17 15:54:54.572140 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.572058 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25a8e17d3a8195e864baab455c283f181b5ebbdb5d574dd1df9531467f18773"} err="failed to get container status \"c25a8e17d3a8195e864baab455c283f181b5ebbdb5d574dd1df9531467f18773\": rpc error: code = NotFound desc = could not find container \"c25a8e17d3a8195e864baab455c283f181b5ebbdb5d574dd1df9531467f18773\": container with ID starting with c25a8e17d3a8195e864baab455c283f181b5ebbdb5d574dd1df9531467f18773 not found: ID does not exist" Sep 17 15:54:54.572140 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.572076 2581 scope.go:117] "RemoveContainer" containerID="104ac2092d0c94a68daddf7e43dfb6007ec55cca436d9f5704d812df9dbd2bba" Sep 17 15:54:54.581631 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.581596 2581 scope.go:117] "RemoveContainer" containerID="104ac2092d0c94a68daddf7e43dfb6007ec55cca436d9f5704d812df9dbd2bba" Sep 17 15:54:54.581975 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:54:54.581957 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104ac2092d0c94a68daddf7e43dfb6007ec55cca436d9f5704d812df9dbd2bba\": container with ID starting with 104ac2092d0c94a68daddf7e43dfb6007ec55cca436d9f5704d812df9dbd2bba not found: ID does not exist" containerID="104ac2092d0c94a68daddf7e43dfb6007ec55cca436d9f5704d812df9dbd2bba" Sep 17 15:54:54.582032 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.581984 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104ac2092d0c94a68daddf7e43dfb6007ec55cca436d9f5704d812df9dbd2bba"} err="failed to get container status \"104ac2092d0c94a68daddf7e43dfb6007ec55cca436d9f5704d812df9dbd2bba\": rpc error: code = NotFound desc = could not find container \"104ac2092d0c94a68daddf7e43dfb6007ec55cca436d9f5704d812df9dbd2bba\": container with ID starting with 104ac2092d0c94a68daddf7e43dfb6007ec55cca436d9f5704d812df9dbd2bba not found: ID does not exist" Sep 17 15:54:54.640971 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.640940 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-dshm\") pod \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " Sep 17 15:54:54.641129 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.640981 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-tmp-dir\") pod \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " Sep 17 15:54:54.641129 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.641044 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-home\") pod \"25424ac6-a827-4f63-abb3-58a873ff27fa\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " Sep 17 15:54:54.641129 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.641081 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z684w\" (UniqueName: \"kubernetes.io/projected/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-kube-api-access-z684w\") pod \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " Sep 17 15:54:54.641284 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.641159 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25424ac6-a827-4f63-abb3-58a873ff27fa-tls-certs\") pod \"25424ac6-a827-4f63-abb3-58a873ff27fa\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " Sep 17 15:54:54.641284 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.641240 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073" (UID: "c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:54:54.641284 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.641272 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-home" (OuterVolumeSpecName: "home") pod "25424ac6-a827-4f63-abb3-58a873ff27fa" (UID: "25424ac6-a827-4f63-abb3-58a873ff27fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:54:54.641445 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.641289 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-tls-certs\") pod \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " Sep 17 15:54:54.641445 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.641327 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-dshm\") pod \"25424ac6-a827-4f63-abb3-58a873ff27fa\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " Sep 17 15:54:54.641445 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.641353 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-model-cache\") pod \"25424ac6-a827-4f63-abb3-58a873ff27fa\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " Sep 17 15:54:54.641445 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.641371 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-model-cache\") pod \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " Sep 17 15:54:54.641445 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.641405 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-home\") pod \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\" (UID: \"c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073\") " Sep 17 15:54:54.641445 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.641444 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-tmp-dir\") pod \"25424ac6-a827-4f63-abb3-58a873ff27fa\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " Sep 17 15:54:54.641761 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.641474 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjt27\" (UniqueName: \"kubernetes.io/projected/25424ac6-a827-4f63-abb3-58a873ff27fa-kube-api-access-xjt27\") pod \"25424ac6-a827-4f63-abb3-58a873ff27fa\" (UID: \"25424ac6-a827-4f63-abb3-58a873ff27fa\") " Sep 17 15:54:54.641840 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.641800 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-home" (OuterVolumeSpecName: "home") pod "c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073" (UID: "c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:54:54.642052 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.642029 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-home\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:54.642052 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.642041 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25424ac6-a827-4f63-abb3-58a873ff27fa" (UID: "25424ac6-a827-4f63-abb3-58a873ff27fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:54:54.642195 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.642059 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-tmp-dir\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:54.642195 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.642074 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-home\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:54.642195 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.642091 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "25424ac6-a827-4f63-abb3-58a873ff27fa" (UID: "25424ac6-a827-4f63-abb3-58a873ff27fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:54:54.642195 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.642180 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073" (UID: "c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:54:54.643479 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.643447 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25424ac6-a827-4f63-abb3-58a873ff27fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25424ac6-a827-4f63-abb3-58a873ff27fa" (UID: "25424ac6-a827-4f63-abb3-58a873ff27fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:54:54.643608 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.643574 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-dshm" (OuterVolumeSpecName: "dshm") pod "c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073" (UID: "c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:54:54.643725 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.643692 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-kube-api-access-z684w" (OuterVolumeSpecName: "kube-api-access-z684w") pod "c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073" (UID: "c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073"). InnerVolumeSpecName "kube-api-access-z684w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:54:54.643850 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.643826 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073" (UID: "c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:54:54.644505 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.644482 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-dshm" (OuterVolumeSpecName: "dshm") pod "25424ac6-a827-4f63-abb3-58a873ff27fa" (UID: "25424ac6-a827-4f63-abb3-58a873ff27fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:54:54.644677 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.644660 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25424ac6-a827-4f63-abb3-58a873ff27fa-kube-api-access-xjt27" (OuterVolumeSpecName: "kube-api-access-xjt27") pod "25424ac6-a827-4f63-abb3-58a873ff27fa" (UID: "25424ac6-a827-4f63-abb3-58a873ff27fa"). InnerVolumeSpecName "kube-api-access-xjt27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:54:54.716969 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.716844 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:54.747434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.743835 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-tls-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:54.747434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.743867 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-dshm\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:54.747434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.743890 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-model-cache\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:54.747434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.743902 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-model-cache\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:54.747434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.743944 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25424ac6-a827-4f63-abb3-58a873ff27fa-tmp-dir\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:54.747434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.743961 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjt27\" (UniqueName: \"kubernetes.io/projected/25424ac6-a827-4f63-abb3-58a873ff27fa-kube-api-access-xjt27\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:54.747434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.743979 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-dshm\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:54.747434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.743993 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z684w\" (UniqueName: \"kubernetes.io/projected/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073-kube-api-access-z684w\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:54.747434 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.744006 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25424ac6-a827-4f63-abb3-58a873ff27fa-tls-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:54:54.885658 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.885631 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh"] Sep 17 15:54:54.893633 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.893605 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/distinct-w0-849b71-kserve-76d76747fb-8hbwh"] Sep 17 15:54:54.915275 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.915249 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws"] Sep 17 15:54:54.917458 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:54.917437 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/simulator-w0-849b71-kserve-f564bdc76-sbsws"] Sep 17 15:54:55.463053 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:55.463018 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25424ac6-a827-4f63-abb3-58a873ff27fa" path="/var/lib/kubelet/pods/25424ac6-a827-4f63-abb3-58a873ff27fa/volumes" Sep 17 15:54:55.463452 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:55.463437 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073" path="/var/lib/kubelet/pods/c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073/volumes" Sep 17 15:54:55.463766 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:55.463754 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0537ab3-dc97-4bd8-adec-f6b1ccdda18e" path="/var/lib/kubelet/pods/e0537ab3-dc97-4bd8-adec-f6b1ccdda18e/volumes" Sep 17 15:54:55.509279 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:55.509251 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" Sep 17 15:54:55.509422 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:55.509390 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" Sep 17 15:54:56.096689 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:56.096658 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:56.553152 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:56.553122 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:57.065875 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:57.065843 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:57.187073 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:57.187043 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:57.478586 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:57.478508 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:57.854548 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:57.854519 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:58.137660 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:58.137586 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:54:59.267141 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:54:59.267098 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:00.111208 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:00.111172 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:01.054829 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:01.054797 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:01.250331 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:01.250294 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:01.370523 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:01.370434 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:01.699824 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:01.699737 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c"] Sep 17 15:55:01.699997 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:01.699967 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" podUID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerName="istio-proxy" containerID="cri-o://4666d88018638d9ddbb2b0b89dc5d999b441951387d8b86e7c58c20b74779d08" gracePeriod=30 Sep 17 15:55:01.752668 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:01.752628 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:02.350655 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:02.350621 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:03.208102 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:03.208065 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:04.107511 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:04.107473 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:04.204594 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:04.204559 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:04.321272 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:04.321222 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:05.056054 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:05.056023 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:05.284781 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:05.284751 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:05.768021 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:05.767990 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:06.832621 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:06.832592 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:06.928878 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:06.928673 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:06.950432 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:06.950407 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:55:07.047084 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047039 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-credential-socket\") pod \"948331da-4b2c-4f6e-bf94-24551fdf86b5\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " Sep 17 15:55:07.047279 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047134 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-workload-certs\") pod \"948331da-4b2c-4f6e-bf94-24551fdf86b5\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " Sep 17 15:55:07.047279 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047182 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948331da-4b2c-4f6e-bf94-24551fdf86b5-istiod-ca-cert\") pod \"948331da-4b2c-4f6e-bf94-24551fdf86b5\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " Sep 17 15:55:07.047279 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047250 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-podinfo\") pod \"948331da-4b2c-4f6e-bf94-24551fdf86b5\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " Sep 17 15:55:07.047452 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047303 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8nmn\" (UniqueName: \"kubernetes.io/projected/948331da-4b2c-4f6e-bf94-24551fdf86b5-kube-api-access-h8nmn\") pod \"948331da-4b2c-4f6e-bf94-24551fdf86b5\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " Sep 17 15:55:07.047452 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047320 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "948331da-4b2c-4f6e-bf94-24551fdf86b5" (UID: "948331da-4b2c-4f6e-bf94-24551fdf86b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:55:07.047452 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047353 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-token\") pod \"948331da-4b2c-4f6e-bf94-24551fdf86b5\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " Sep 17 15:55:07.047452 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047384 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-data\") pod \"948331da-4b2c-4f6e-bf94-24551fdf86b5\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " Sep 17 15:55:07.047452 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047412 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-workload-socket\") pod \"948331da-4b2c-4f6e-bf94-24551fdf86b5\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " Sep 17 15:55:07.047712 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047454 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "948331da-4b2c-4f6e-bf94-24551fdf86b5" (UID: "948331da-4b2c-4f6e-bf94-24551fdf86b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:55:07.047712 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047472 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-envoy\") pod \"948331da-4b2c-4f6e-bf94-24551fdf86b5\" (UID: \"948331da-4b2c-4f6e-bf94-24551fdf86b5\") " Sep 17 15:55:07.047712 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047667 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-data" (OuterVolumeSpecName: "istio-data") pod "948331da-4b2c-4f6e-bf94-24551fdf86b5" (UID: "948331da-4b2c-4f6e-bf94-24551fdf86b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:55:07.047865 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047697 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948331da-4b2c-4f6e-bf94-24551fdf86b5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "948331da-4b2c-4f6e-bf94-24551fdf86b5" (UID: "948331da-4b2c-4f6e-bf94-24551fdf86b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:55:07.047865 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047814 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-data\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:07.047865 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047834 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-credential-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:07.047865 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047847 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-workload-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:07.047865 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.047861 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948331da-4b2c-4f6e-bf94-24551fdf86b5-istiod-ca-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:07.048161 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.048021 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "948331da-4b2c-4f6e-bf94-24551fdf86b5" (UID: "948331da-4b2c-4f6e-bf94-24551fdf86b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:55:07.050485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.050416 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948331da-4b2c-4f6e-bf94-24551fdf86b5-kube-api-access-h8nmn" (OuterVolumeSpecName: "kube-api-access-h8nmn") pod "948331da-4b2c-4f6e-bf94-24551fdf86b5" (UID: "948331da-4b2c-4f6e-bf94-24551fdf86b5"). InnerVolumeSpecName "kube-api-access-h8nmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:55:07.050485 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.050426 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "948331da-4b2c-4f6e-bf94-24551fdf86b5" (UID: "948331da-4b2c-4f6e-bf94-24551fdf86b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:55:07.050827 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.050799 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "948331da-4b2c-4f6e-bf94-24551fdf86b5" (UID: "948331da-4b2c-4f6e-bf94-24551fdf86b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:55:07.051056 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.051029 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "948331da-4b2c-4f6e-bf94-24551fdf86b5" (UID: "948331da-4b2c-4f6e-bf94-24551fdf86b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 15:55:07.063633 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.063600 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:07.149294 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.149258 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-workload-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:07.149294 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.149286 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-envoy\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:07.149294 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.149300 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-podinfo\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:07.149524 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.149313 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8nmn\" (UniqueName: \"kubernetes.io/projected/948331da-4b2c-4f6e-bf94-24551fdf86b5-kube-api-access-h8nmn\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:07.149524 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.149325 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948331da-4b2c-4f6e-bf94-24551fdf86b5-istio-token\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:07.455310 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.455231 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:07.559753 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.559726 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:07.564813 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.564789 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67"] Sep 17 15:55:07.565189 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565177 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerName="istio-proxy" Sep 17 15:55:07.565237 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565192 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerName="istio-proxy" Sep 17 15:55:07.565237 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565214 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerName="istio-proxy" Sep 17 15:55:07.565237 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565220 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerName="istio-proxy" Sep 17 15:55:07.565237 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565232 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25424ac6-a827-4f63-abb3-58a873ff27fa" containerName="main" Sep 17 15:55:07.565357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565240 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="25424ac6-a827-4f63-abb3-58a873ff27fa" containerName="main" Sep 17 15:55:07.565357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565249 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0537ab3-dc97-4bd8-adec-f6b1ccdda18e" containerName="payload-pre-processing" Sep 17 15:55:07.565357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565255 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0537ab3-dc97-4bd8-adec-f6b1ccdda18e" containerName="payload-pre-processing" Sep 17 15:55:07.565357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565262 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073" containerName="main" Sep 17 15:55:07.565357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565267 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073" containerName="main" Sep 17 15:55:07.565357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565332 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerName="istio-proxy" Sep 17 15:55:07.565357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565341 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0537ab3-dc97-4bd8-adec-f6b1ccdda18e" containerName="payload-pre-processing" Sep 17 15:55:07.565357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565346 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8dab90a-6fbd-47aa-9c6a-04bf4ad2b073" containerName="main" Sep 17 15:55:07.565357 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.565351 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="25424ac6-a827-4f63-abb3-58a873ff27fa" containerName="main" Sep 17 15:55:07.568605 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.568586 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" Sep 17 15:55:07.574204 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.574176 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8f99a1\"" Sep 17 15:55:07.580937 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.580884 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt"] Sep 17 15:55:07.581635 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.581612 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerName="istio-proxy" Sep 17 15:55:07.584996 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.584967 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" Sep 17 15:55:07.587950 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.587676 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8f99a1-dockercfg-8dsrj\"" Sep 17 15:55:07.610698 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.610668 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt"] Sep 17 15:55:07.638786 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.638753 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67"] Sep 17 15:55:07.643047 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.643015 2581 generic.go:356] "Generic (PLEG): container finished" podID="948331da-4b2c-4f6e-bf94-24551fdf86b5" containerID="4666d88018638d9ddbb2b0b89dc5d999b441951387d8b86e7c58c20b74779d08" exitCode=0 Sep 17 15:55:07.643230 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.643209 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" event={"ID":"948331da-4b2c-4f6e-bf94-24551fdf86b5","Type":"ContainerDied","Data":"4666d88018638d9ddbb2b0b89dc5d999b441951387d8b86e7c58c20b74779d08"} Sep 17 15:55:07.643286 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.643269 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" event={"ID":"948331da-4b2c-4f6e-bf94-24551fdf86b5","Type":"ContainerDied","Data":"ba8fea65d52abb4558f85cc00f9842c9e534abc6e4cafacd70b2556a49b433f4"} Sep 17 15:55:07.643286 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.643276 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c" Sep 17 15:55:07.643364 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.643302 2581 scope.go:117] "RemoveContainer" containerID="4666d88018638d9ddbb2b0b89dc5d999b441951387d8b86e7c58c20b74779d08" Sep 17 15:55:07.653978 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.653959 2581 scope.go:117] "RemoveContainer" containerID="99c8b3876b11686a643bb7e1e086d91921c1364a53f09abd066149b206809734" Sep 17 15:55:07.654073 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.654052 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c2mm\" (UniqueName: \"kubernetes.io/projected/f75059c4-89b5-42f4-afba-4b454cf37500-kube-api-access-2c2mm\") pod \"payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67\" (UID: \"f75059c4-89b5-42f4-afba-4b454cf37500\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" Sep 17 15:55:07.654132 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.654092 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75059c4-89b5-42f4-afba-4b454cf37500-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67\" (UID: \"f75059c4-89b5-42f4-afba-4b454cf37500\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" Sep 17 15:55:07.654231 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.654205 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4266c95-b4a2-4760-bdb3-ebdb38db785a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt\" (UID: \"f4266c95-b4a2-4760-bdb3-ebdb38db785a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" Sep 17 15:55:07.654373 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.654334 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwh7g\" (UniqueName: \"kubernetes.io/projected/f4266c95-b4a2-4760-bdb3-ebdb38db785a-kube-api-access-mwh7g\") pod \"payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt\" (UID: \"f4266c95-b4a2-4760-bdb3-ebdb38db785a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" Sep 17 15:55:07.662424 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.662394 2581 scope.go:117] "RemoveContainer" containerID="4666d88018638d9ddbb2b0b89dc5d999b441951387d8b86e7c58c20b74779d08" Sep 17 15:55:07.662710 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:55:07.662693 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4666d88018638d9ddbb2b0b89dc5d999b441951387d8b86e7c58c20b74779d08\": container with ID starting with 4666d88018638d9ddbb2b0b89dc5d999b441951387d8b86e7c58c20b74779d08 not found: ID does not exist" containerID="4666d88018638d9ddbb2b0b89dc5d999b441951387d8b86e7c58c20b74779d08" Sep 17 15:55:07.662767 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.662718 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4666d88018638d9ddbb2b0b89dc5d999b441951387d8b86e7c58c20b74779d08"} err="failed to get container status \"4666d88018638d9ddbb2b0b89dc5d999b441951387d8b86e7c58c20b74779d08\": rpc error: code = NotFound desc = could not find container \"4666d88018638d9ddbb2b0b89dc5d999b441951387d8b86e7c58c20b74779d08\": container with ID starting with 4666d88018638d9ddbb2b0b89dc5d999b441951387d8b86e7c58c20b74779d08 not found: ID does not exist" Sep 17 15:55:07.662767 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.662734 2581 scope.go:117] "RemoveContainer" containerID="99c8b3876b11686a643bb7e1e086d91921c1364a53f09abd066149b206809734" Sep 17 15:55:07.663038 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:55:07.663021 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c8b3876b11686a643bb7e1e086d91921c1364a53f09abd066149b206809734\": container with ID starting with 99c8b3876b11686a643bb7e1e086d91921c1364a53f09abd066149b206809734 not found: ID does not exist" containerID="99c8b3876b11686a643bb7e1e086d91921c1364a53f09abd066149b206809734" Sep 17 15:55:07.663087 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.663044 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c8b3876b11686a643bb7e1e086d91921c1364a53f09abd066149b206809734"} err="failed to get container status \"99c8b3876b11686a643bb7e1e086d91921c1364a53f09abd066149b206809734\": rpc error: code = NotFound desc = could not find container \"99c8b3876b11686a643bb7e1e086d91921c1364a53f09abd066149b206809734\": container with ID starting with 99c8b3876b11686a643bb7e1e086d91921c1364a53f09abd066149b206809734 not found: ID does not exist" Sep 17 15:55:07.675470 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.674394 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c"] Sep 17 15:55:07.688791 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.688755 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-849b71-openshift-default-77697b697d-9sn6c"] Sep 17 15:55:07.755025 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.754943 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh7g\" (UniqueName: \"kubernetes.io/projected/f4266c95-b4a2-4760-bdb3-ebdb38db785a-kube-api-access-mwh7g\") pod \"payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt\" (UID: \"f4266c95-b4a2-4760-bdb3-ebdb38db785a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" Sep 17 15:55:07.755025 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.754983 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c2mm\" (UniqueName: \"kubernetes.io/projected/f75059c4-89b5-42f4-afba-4b454cf37500-kube-api-access-2c2mm\") pod \"payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67\" (UID: \"f75059c4-89b5-42f4-afba-4b454cf37500\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" Sep 17 15:55:07.755226 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.755100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75059c4-89b5-42f4-afba-4b454cf37500-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67\" (UID: \"f75059c4-89b5-42f4-afba-4b454cf37500\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" Sep 17 15:55:07.755226 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.755179 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4266c95-b4a2-4760-bdb3-ebdb38db785a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt\" (UID: \"f4266c95-b4a2-4760-bdb3-ebdb38db785a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" Sep 17 15:55:07.755794 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.755776 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75059c4-89b5-42f4-afba-4b454cf37500-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67\" (UID: \"f75059c4-89b5-42f4-afba-4b454cf37500\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" Sep 17 15:55:07.755837 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.755806 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4266c95-b4a2-4760-bdb3-ebdb38db785a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt\" (UID: \"f4266c95-b4a2-4760-bdb3-ebdb38db785a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" Sep 17 15:55:07.770656 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.770628 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c2mm\" (UniqueName: \"kubernetes.io/projected/f75059c4-89b5-42f4-afba-4b454cf37500-kube-api-access-2c2mm\") pod \"payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67\" (UID: \"f75059c4-89b5-42f4-afba-4b454cf37500\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" Sep 17 15:55:07.770779 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.770684 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwh7g\" (UniqueName: \"kubernetes.io/projected/f4266c95-b4a2-4760-bdb3-ebdb38db785a-kube-api-access-mwh7g\") pod \"payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt\" (UID: \"f4266c95-b4a2-4760-bdb3-ebdb38db785a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" Sep 17 15:55:07.891961 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.891876 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" Sep 17 15:55:07.910992 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.910960 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" Sep 17 15:55:07.935349 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:07.935306 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:08.188053 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:08.188011 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67"] Sep 17 15:55:08.224177 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:08.224148 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt"] Sep 17 15:55:08.234024 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:55:08.234001 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4266c95_b4a2_4760_bdb3_ebdb38db785a.slice/crio-a204814f61db1f063e8188d45313587cda7dd84c68c7545920aca1bf4c2d9ae5 WatchSource:0}: Error finding container a204814f61db1f063e8188d45313587cda7dd84c68c7545920aca1bf4c2d9ae5: Status 404 returned error can't find the container with id a204814f61db1f063e8188d45313587cda7dd84c68c7545920aca1bf4c2d9ae5 Sep 17 15:55:08.651665 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:08.651629 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" event={"ID":"f75059c4-89b5-42f4-afba-4b454cf37500","Type":"ContainerStarted","Data":"1f253fbdcfd9555618dca1f424afc323deb4b70a14b2d97881d51864fcce2d4f"} Sep 17 15:55:08.651853 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:08.651721 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" event={"ID":"f75059c4-89b5-42f4-afba-4b454cf37500","Type":"ContainerStarted","Data":"8f4c6f2617c549c94f5c425198e1535debe91566481d0182e019b12a3f79046b"} Sep 17 15:55:08.652004 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:08.651984 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" Sep 17 15:55:08.653574 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:08.653551 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" event={"ID":"f4266c95-b4a2-4760-bdb3-ebdb38db785a","Type":"ContainerStarted","Data":"e013bfb95fc16402c7e094f6106b688734e0d42e33a1751aa49c9f6a72defe16"} Sep 17 15:55:08.653686 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:08.653585 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" event={"ID":"f4266c95-b4a2-4760-bdb3-ebdb38db785a","Type":"ContainerStarted","Data":"a204814f61db1f063e8188d45313587cda7dd84c68c7545920aca1bf4c2d9ae5"} Sep 17 15:55:08.653747 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:08.653699 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" Sep 17 15:55:08.671884 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:08.671827 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" podStartSLOduration=1.671812888 podStartE2EDuration="1.671812888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:08.669863345 +0000 UTC m=+1495.666100091" watchObservedRunningTime="2026-09-17 15:55:08.671812888 +0000 UTC m=+1495.668049634" Sep 17 15:55:08.693139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:08.693093 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" podStartSLOduration=1.6930754609999998 podStartE2EDuration="1.693075461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:08.689243554 +0000 UTC m=+1495.685480300" watchObservedRunningTime="2026-09-17 15:55:08.693075461 +0000 UTC m=+1495.689312208" Sep 17 15:55:09.462107 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:09.462070 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="948331da-4b2c-4f6e-bf94-24551fdf86b5" path="/var/lib/kubelet/pods/948331da-4b2c-4f6e-bf94-24551fdf86b5/volumes" Sep 17 15:55:09.506311 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:09.506269 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:12.154701 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:12.154668 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:15.694035 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:15.694006 2581 generic.go:356] "Generic (PLEG): container finished" podID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerID="fc9d5068c27ab4d61bc489df8dfe0cdc53c99a4ca97f66d90ed22f0473093414" exitCode=0 Sep 17 15:55:15.694352 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:15.694078 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" event={"ID":"82906d2d-5c60-4e02-829f-d010c89a70f1","Type":"ContainerDied","Data":"fc9d5068c27ab4d61bc489df8dfe0cdc53c99a4ca97f66d90ed22f0473093414"} Sep 17 15:55:15.694488 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:15.694475 2581 scope.go:117] "RemoveContainer" containerID="fc9d5068c27ab4d61bc489df8dfe0cdc53c99a4ca97f66d90ed22f0473093414" Sep 17 15:55:15.695568 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:15.695539 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:55:15.695642 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:15.695608 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:55:15.695642 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:15.695638 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:55:16.408036 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:16.407991 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:16.502573 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:16.502508 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:16.705365 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:16.705284 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" event={"ID":"82906d2d-5c60-4e02-829f-d010c89a70f1","Type":"ContainerStarted","Data":"d45d4cc4bb078fe7ddea5da54407f1460f8d0b95e30607c32f092d460285eef9"} Sep 17 15:55:17.061241 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:17.061205 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:55:17.062396 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:17.062367 2581 patch_prober.go:28] interesting pod/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 17 15:55:17.062511 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:17.062422 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" podUID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 17 15:55:17.965831 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:17.965800 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:18.060716 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:18.060679 2581 patch_prober.go:28] interesting pod/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 17 15:55:18.060893 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:18.060734 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" podUID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 17 15:55:19.060659 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:19.060626 2581 patch_prober.go:28] interesting pod/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 17 15:55:19.061062 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:19.060682 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" podUID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 17 15:55:19.661093 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:19.661065 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" Sep 17 15:55:19.661320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:19.661129 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" Sep 17 15:55:19.981834 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:19.981736 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:20.061332 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:20.061302 2581 patch_prober.go:28] interesting pod/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 17 15:55:20.061692 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:20.061351 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" podUID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 17 15:55:20.432305 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:20.432268 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:55:21.061410 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:21.061375 2581 patch_prober.go:28] interesting pod/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Sep 17 15:55:21.061782 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:21.061429 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" podUID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Sep 17 15:55:22.065402 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:22.065359 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:55:22.066549 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:22.066528 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:55:24.674257 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:24.674220 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:28.016212 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:28.016177 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:28.117156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:28.117124 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:29.491331 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:29.491295 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:30.374443 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:30.374409 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:30.721387 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:30.721297 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:31.253234 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:31.253199 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:31.641840 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:31.641804 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:32.520260 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:32.520223 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:33.373641 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:33.373611 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:34.126750 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:34.126711 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:34.366937 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:34.366879 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68"] Sep 17 15:55:34.370455 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:34.370434 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68" Sep 17 15:55:34.373047 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:34.373028 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\"" Sep 17 15:55:34.373254 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:34.373240 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 15:55:34.383298 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:34.383240 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68"] Sep 17 15:55:34.509065 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:34.509033 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bff5cd9-bfad-4610-822f-089333ffc319-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c686d0-2jk68\" (UID: \"9bff5cd9-bfad-4610-822f-089333ffc319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68" Sep 17 15:55:34.610414 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:34.610379 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bff5cd9-bfad-4610-822f-089333ffc319-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c686d0-2jk68\" (UID: \"9bff5cd9-bfad-4610-822f-089333ffc319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68" Sep 17 15:55:34.611001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:34.610985 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bff5cd9-bfad-4610-822f-089333ffc319-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c686d0-2jk68\" (UID: \"9bff5cd9-bfad-4610-822f-089333ffc319\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68" Sep 17 15:55:34.681718 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:34.681647 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68" Sep 17 15:55:34.955373 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:34.955339 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68"] Sep 17 15:55:34.956264 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:55:34.956238 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bff5cd9_bfad_4610_822f_089333ffc319.slice/crio-d4744bd1886e2b177bc94825cfaefb84b107dca3238514e4073467420ccd6626 WatchSource:0}: Error finding container d4744bd1886e2b177bc94825cfaefb84b107dca3238514e4073467420ccd6626: Status 404 returned error can't find the container with id d4744bd1886e2b177bc94825cfaefb84b107dca3238514e4073467420ccd6626 Sep 17 15:55:35.202524 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:35.202499 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:35.814561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:35.814529 2581 generic.go:356] "Generic (PLEG): container finished" podID="9bff5cd9-bfad-4610-822f-089333ffc319" containerID="ba73a027b08a3fa967ebe73ce18df307280df716e58e9e9d0c6189ad245c3d9e" exitCode=0 Sep 17 15:55:35.814762 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:35.814578 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68" event={"ID":"9bff5cd9-bfad-4610-822f-089333ffc319","Type":"ContainerDied","Data":"ba73a027b08a3fa967ebe73ce18df307280df716e58e9e9d0c6189ad245c3d9e"} Sep 17 15:55:35.814762 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:35.814614 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68" event={"ID":"9bff5cd9-bfad-4610-822f-089333ffc319","Type":"ContainerStarted","Data":"d4744bd1886e2b177bc94825cfaefb84b107dca3238514e4073467420ccd6626"} Sep 17 15:55:36.052204 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:36.052176 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:36.809427 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:36.809368 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:36.956643 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:36.956621 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68" Sep 17 15:55:37.132335 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:37.132254 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bff5cd9-bfad-4610-822f-089333ffc319-maas-api-service-ca\") pod \"9bff5cd9-bfad-4610-822f-089333ffc319\" (UID: \"9bff5cd9-bfad-4610-822f-089333ffc319\") " Sep 17 15:55:37.132613 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:37.132589 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bff5cd9-bfad-4610-822f-089333ffc319-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9bff5cd9-bfad-4610-822f-089333ffc319" (UID: "9bff5cd9-bfad-4610-822f-089333ffc319"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:55:37.233138 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:37.233104 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bff5cd9-bfad-4610-822f-089333ffc319-maas-api-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:37.807539 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:37.807508 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:37.828881 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:37.828857 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68" Sep 17 15:55:37.829249 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:37.828889 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68" event={"ID":"9bff5cd9-bfad-4610-822f-089333ffc319","Type":"ContainerDied","Data":"d4744bd1886e2b177bc94825cfaefb84b107dca3238514e4073467420ccd6626"} Sep 17 15:55:37.829249 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:37.828946 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4744bd1886e2b177bc94825cfaefb84b107dca3238514e4073467420ccd6626" Sep 17 15:55:38.594269 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:38.594236 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:39.472091 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:39.472058 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:40.425560 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:40.425526 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:44.575264 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:44.575233 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:50.403351 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.403319 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:50.458882 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.458845 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx"] Sep 17 15:55:50.459164 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.459139 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" podUID="cca909d6-fef7-4bdd-b472-e83fbe704f62" containerName="payload-processing" containerID="cri-o://90c51eb62548adb69e54b6bff3381fc361c76abf11b376d3522e8d0b6996773a" gracePeriod=30 Sep 17 15:55:50.468492 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.468464 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5"] Sep 17 15:55:50.468683 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.468649 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" podUID="e088820e-fecb-45a1-8883-c58802b693f0" containerName="payload-pre-processing" containerID="cri-o://ab33c87b41dd03e55eb3c0f2f96db6fe2b14b76a06cef5234be824afa4d37a51" gracePeriod=30 Sep 17 15:55:50.525399 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.525364 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:50.670888 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.670860 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:50.731824 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.731804 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" Sep 17 15:55:50.733475 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.733424 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68"] Sep 17 15:55:50.737213 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.737187 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c686d0-2jk68"] Sep 17 15:55:50.746957 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.746940 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" Sep 17 15:55:50.749252 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.749236 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km266\" (UniqueName: \"kubernetes.io/projected/cca909d6-fef7-4bdd-b472-e83fbe704f62-kube-api-access-km266\") pod \"cca909d6-fef7-4bdd-b472-e83fbe704f62\" (UID: \"cca909d6-fef7-4bdd-b472-e83fbe704f62\") " Sep 17 15:55:50.749320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.749266 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j8p8\" (UniqueName: \"kubernetes.io/projected/e088820e-fecb-45a1-8883-c58802b693f0-kube-api-access-7j8p8\") pod \"e088820e-fecb-45a1-8883-c58802b693f0\" (UID: \"e088820e-fecb-45a1-8883-c58802b693f0\") " Sep 17 15:55:50.749320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.749294 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e088820e-fecb-45a1-8883-c58802b693f0-plugins-config-volume\") pod \"e088820e-fecb-45a1-8883-c58802b693f0\" (UID: \"e088820e-fecb-45a1-8883-c58802b693f0\") " Sep 17 15:55:50.749408 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.749319 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca909d6-fef7-4bdd-b472-e83fbe704f62-plugins-config-volume\") pod \"cca909d6-fef7-4bdd-b472-e83fbe704f62\" (UID: \"cca909d6-fef7-4bdd-b472-e83fbe704f62\") " Sep 17 15:55:50.749656 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.749635 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e088820e-fecb-45a1-8883-c58802b693f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e088820e-fecb-45a1-8883-c58802b693f0" (UID: "e088820e-fecb-45a1-8883-c58802b693f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:55:50.749722 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.749668 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca909d6-fef7-4bdd-b472-e83fbe704f62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cca909d6-fef7-4bdd-b472-e83fbe704f62" (UID: "cca909d6-fef7-4bdd-b472-e83fbe704f62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:55:50.751247 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.751228 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca909d6-fef7-4bdd-b472-e83fbe704f62-kube-api-access-km266" (OuterVolumeSpecName: "kube-api-access-km266") pod "cca909d6-fef7-4bdd-b472-e83fbe704f62" (UID: "cca909d6-fef7-4bdd-b472-e83fbe704f62"). InnerVolumeSpecName "kube-api-access-km266". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:55:50.751311 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.751269 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e088820e-fecb-45a1-8883-c58802b693f0-kube-api-access-7j8p8" (OuterVolumeSpecName: "kube-api-access-7j8p8") pod "e088820e-fecb-45a1-8883-c58802b693f0" (UID: "e088820e-fecb-45a1-8883-c58802b693f0"). InnerVolumeSpecName "kube-api-access-7j8p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:55:50.767491 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.767467 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:50.850395 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.850360 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km266\" (UniqueName: \"kubernetes.io/projected/cca909d6-fef7-4bdd-b472-e83fbe704f62-kube-api-access-km266\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:50.850395 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.850389 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j8p8\" (UniqueName: \"kubernetes.io/projected/e088820e-fecb-45a1-8883-c58802b693f0-kube-api-access-7j8p8\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:50.850395 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.850400 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e088820e-fecb-45a1-8883-c58802b693f0-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:50.850787 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.850409 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca909d6-fef7-4bdd-b472-e83fbe704f62-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:55:50.900951 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.900903 2581 generic.go:356] "Generic (PLEG): container finished" podID="e088820e-fecb-45a1-8883-c58802b693f0" containerID="ab33c87b41dd03e55eb3c0f2f96db6fe2b14b76a06cef5234be824afa4d37a51" exitCode=0 Sep 17 15:55:50.901135 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.900979 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" Sep 17 15:55:50.901135 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.900986 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" event={"ID":"e088820e-fecb-45a1-8883-c58802b693f0","Type":"ContainerDied","Data":"ab33c87b41dd03e55eb3c0f2f96db6fe2b14b76a06cef5234be824afa4d37a51"} Sep 17 15:55:50.901135 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.901107 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5" event={"ID":"e088820e-fecb-45a1-8883-c58802b693f0","Type":"ContainerDied","Data":"bdeb4eb5d24139b1a3ef6ea3d92db08e3ffa2a5dc60374ef5ca8e2a303d3617f"} Sep 17 15:55:50.901135 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.901127 2581 scope.go:117] "RemoveContainer" containerID="ab33c87b41dd03e55eb3c0f2f96db6fe2b14b76a06cef5234be824afa4d37a51" Sep 17 15:55:50.902506 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.902477 2581 generic.go:356] "Generic (PLEG): container finished" podID="cca909d6-fef7-4bdd-b472-e83fbe704f62" containerID="90c51eb62548adb69e54b6bff3381fc361c76abf11b376d3522e8d0b6996773a" exitCode=0 Sep 17 15:55:50.902622 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.902520 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" Sep 17 15:55:50.902622 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.902533 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" event={"ID":"cca909d6-fef7-4bdd-b472-e83fbe704f62","Type":"ContainerDied","Data":"90c51eb62548adb69e54b6bff3381fc361c76abf11b376d3522e8d0b6996773a"} Sep 17 15:55:50.902622 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.902569 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx" event={"ID":"cca909d6-fef7-4bdd-b472-e83fbe704f62","Type":"ContainerDied","Data":"669ad78c16fe0b87f851cc9487fe1c6c9f2d3cebc897bc3167dcbd1cf1e0e5a7"} Sep 17 15:55:50.911862 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.911844 2581 scope.go:117] "RemoveContainer" containerID="ab33c87b41dd03e55eb3c0f2f96db6fe2b14b76a06cef5234be824afa4d37a51" Sep 17 15:55:50.912192 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:55:50.912171 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab33c87b41dd03e55eb3c0f2f96db6fe2b14b76a06cef5234be824afa4d37a51\": container with ID starting with ab33c87b41dd03e55eb3c0f2f96db6fe2b14b76a06cef5234be824afa4d37a51 not found: ID does not exist" containerID="ab33c87b41dd03e55eb3c0f2f96db6fe2b14b76a06cef5234be824afa4d37a51" Sep 17 15:55:50.912276 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.912201 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab33c87b41dd03e55eb3c0f2f96db6fe2b14b76a06cef5234be824afa4d37a51"} err="failed to get container status \"ab33c87b41dd03e55eb3c0f2f96db6fe2b14b76a06cef5234be824afa4d37a51\": rpc error: code = NotFound desc = could not find container \"ab33c87b41dd03e55eb3c0f2f96db6fe2b14b76a06cef5234be824afa4d37a51\": container with ID starting with ab33c87b41dd03e55eb3c0f2f96db6fe2b14b76a06cef5234be824afa4d37a51 not found: ID does not exist" Sep 17 15:55:50.912276 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.912223 2581 scope.go:117] "RemoveContainer" containerID="90c51eb62548adb69e54b6bff3381fc361c76abf11b376d3522e8d0b6996773a" Sep 17 15:55:50.923755 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.922492 2581 scope.go:117] "RemoveContainer" containerID="90c51eb62548adb69e54b6bff3381fc361c76abf11b376d3522e8d0b6996773a" Sep 17 15:55:50.924208 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:55:50.924190 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c51eb62548adb69e54b6bff3381fc361c76abf11b376d3522e8d0b6996773a\": container with ID starting with 90c51eb62548adb69e54b6bff3381fc361c76abf11b376d3522e8d0b6996773a not found: ID does not exist" containerID="90c51eb62548adb69e54b6bff3381fc361c76abf11b376d3522e8d0b6996773a" Sep 17 15:55:50.924271 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.924217 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c51eb62548adb69e54b6bff3381fc361c76abf11b376d3522e8d0b6996773a"} err="failed to get container status \"90c51eb62548adb69e54b6bff3381fc361c76abf11b376d3522e8d0b6996773a\": rpc error: code = NotFound desc = could not find container \"90c51eb62548adb69e54b6bff3381fc361c76abf11b376d3522e8d0b6996773a\": container with ID starting with 90c51eb62548adb69e54b6bff3381fc361c76abf11b376d3522e8d0b6996773a not found: ID does not exist" Sep 17 15:55:50.934161 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.934130 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5"] Sep 17 15:55:50.938950 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.938923 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c686d0-795fb6fbf4-89dv5"] Sep 17 15:55:50.958229 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.958196 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx"] Sep 17 15:55:50.962339 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:50.962312 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c686d0-6cb5f97c97-c7lfx"] Sep 17 15:55:51.107317 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:51.107272 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:51.208314 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:51.208170 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:51.461844 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:51.461766 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bff5cd9-bfad-4610-822f-089333ffc319" path="/var/lib/kubelet/pods/9bff5cd9-bfad-4610-822f-089333ffc319/volumes" Sep 17 15:55:51.462213 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:51.462178 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cca909d6-fef7-4bdd-b472-e83fbe704f62" path="/var/lib/kubelet/pods/cca909d6-fef7-4bdd-b472-e83fbe704f62/volumes" Sep 17 15:55:51.462495 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:51.462481 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e088820e-fecb-45a1-8883-c58802b693f0" path="/var/lib/kubelet/pods/e088820e-fecb-45a1-8883-c58802b693f0/volumes" Sep 17 15:55:52.114371 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:52.114341 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:53.084444 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:53.084407 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:54.252747 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:54.252705 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:55.100454 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:55.100413 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:55.883759 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:55.883724 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:57.965615 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:57.965567 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:58.218400 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.218307 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:58.365961 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.365906 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw"] Sep 17 15:55:58.366219 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.366196 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" podUID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerName="istio-proxy" containerID="cri-o://d45d4cc4bb078fe7ddea5da54407f1460f8d0b95e30607c32f092d460285eef9" gracePeriod=30 Sep 17 15:55:58.462599 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.462569 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:58.762101 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.762069 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb"] Sep 17 15:55:58.762460 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.762448 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e088820e-fecb-45a1-8883-c58802b693f0" containerName="payload-pre-processing" Sep 17 15:55:58.762504 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.762462 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e088820e-fecb-45a1-8883-c58802b693f0" containerName="payload-pre-processing" Sep 17 15:55:58.762504 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.762470 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca909d6-fef7-4bdd-b472-e83fbe704f62" containerName="payload-processing" Sep 17 15:55:58.762504 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.762476 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca909d6-fef7-4bdd-b472-e83fbe704f62" containerName="payload-processing" Sep 17 15:55:58.762504 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.762500 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bff5cd9-bfad-4610-822f-089333ffc319" containerName="revoke-keys" Sep 17 15:55:58.762504 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.762506 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bff5cd9-bfad-4610-822f-089333ffc319" containerName="revoke-keys" Sep 17 15:55:58.762667 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.762564 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e088820e-fecb-45a1-8883-c58802b693f0" containerName="payload-pre-processing" Sep 17 15:55:58.762667 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.762575 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bff5cd9-bfad-4610-822f-089333ffc319" containerName="revoke-keys" Sep 17 15:55:58.762667 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.762581 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cca909d6-fef7-4bdd-b472-e83fbe704f62" containerName="payload-processing" Sep 17 15:55:58.767156 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.767140 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb" Sep 17 15:55:58.772544 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.772525 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 15:55:58.773561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.773430 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\"" Sep 17 15:55:58.785666 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.785635 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb"] Sep 17 15:55:58.826695 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.826670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67776141-3c2d-4b85-bbda-b6800ad3dba8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb\" (UID: \"67776141-3c2d-4b85-bbda-b6800ad3dba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb" Sep 17 15:55:58.928087 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.928053 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67776141-3c2d-4b85-bbda-b6800ad3dba8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb\" (UID: \"67776141-3c2d-4b85-bbda-b6800ad3dba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb" Sep 17 15:55:58.928682 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:58.928661 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67776141-3c2d-4b85-bbda-b6800ad3dba8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb\" (UID: \"67776141-3c2d-4b85-bbda-b6800ad3dba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb" Sep 17 15:55:59.076886 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:59.076859 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb" Sep 17 15:55:59.207163 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:59.207132 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:55:59.361203 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:59.361178 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb"] Sep 17 15:55:59.363267 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:55:59.363242 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67776141_3c2d_4b85_bbda_b6800ad3dba8.slice/crio-6570bdf9a8728995862f1ab21cf0860a0f57f80672a15e84718fe1ce1f03dbba WatchSource:0}: Error finding container 6570bdf9a8728995862f1ab21cf0860a0f57f80672a15e84718fe1ce1f03dbba: Status 404 returned error can't find the container with id 6570bdf9a8728995862f1ab21cf0860a0f57f80672a15e84718fe1ce1f03dbba Sep 17 15:55:59.952387 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:59.952282 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb" event={"ID":"67776141-3c2d-4b85-bbda-b6800ad3dba8","Type":"ContainerStarted","Data":"c126beef6e3eb657ca035ccd2da8cd45b1a2423233a3807311be5205185ca5a4"} Sep 17 15:55:59.952387 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:59.952345 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb" event={"ID":"67776141-3c2d-4b85-bbda-b6800ad3dba8","Type":"ContainerStarted","Data":"6570bdf9a8728995862f1ab21cf0860a0f57f80672a15e84718fe1ce1f03dbba"} Sep 17 15:55:59.972048 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:55:59.972003 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb" podStartSLOduration=1.971986182 podStartE2EDuration="1.971986182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:59.971119726 +0000 UTC m=+1546.967356471" watchObservedRunningTime="2026-09-17 15:55:59.971986182 +0000 UTC m=+1546.968222927" Sep 17 15:56:00.218749 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:00.218662 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:00.959825 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:00.959796 2581 generic.go:356] "Generic (PLEG): container finished" podID="67776141-3c2d-4b85-bbda-b6800ad3dba8" containerID="c126beef6e3eb657ca035ccd2da8cd45b1a2423233a3807311be5205185ca5a4" exitCode=0 Sep 17 15:56:00.960002 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:00.959878 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb" event={"ID":"67776141-3c2d-4b85-bbda-b6800ad3dba8","Type":"ContainerDied","Data":"c126beef6e3eb657ca035ccd2da8cd45b1a2423233a3807311be5205185ca5a4"} Sep 17 15:56:01.104706 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:01.104671 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:02.093108 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:02.093074 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb" Sep 17 15:56:02.114621 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:02.114589 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:02.156819 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:02.156783 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67776141-3c2d-4b85-bbda-b6800ad3dba8-maas-api-service-ca\") pod \"67776141-3c2d-4b85-bbda-b6800ad3dba8\" (UID: \"67776141-3c2d-4b85-bbda-b6800ad3dba8\") " Sep 17 15:56:02.157113 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:02.157089 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67776141-3c2d-4b85-bbda-b6800ad3dba8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67776141-3c2d-4b85-bbda-b6800ad3dba8" (UID: "67776141-3c2d-4b85-bbda-b6800ad3dba8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:56:02.157200 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:02.157188 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67776141-3c2d-4b85-bbda-b6800ad3dba8-maas-api-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:02.972470 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:02.972439 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb" Sep 17 15:56:02.972667 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:02.972478 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb" event={"ID":"67776141-3c2d-4b85-bbda-b6800ad3dba8","Type":"ContainerDied","Data":"6570bdf9a8728995862f1ab21cf0860a0f57f80672a15e84718fe1ce1f03dbba"} Sep 17 15:56:02.972667 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:02.972522 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6570bdf9a8728995862f1ab21cf0860a0f57f80672a15e84718fe1ce1f03dbba" Sep 17 15:56:03.315150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.315110 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:03.627826 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.627800 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:56:03.670166 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670139 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-token\") pod \"82906d2d-5c60-4e02-829f-d010c89a70f1\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " Sep 17 15:56:03.670295 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670188 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-data\") pod \"82906d2d-5c60-4e02-829f-d010c89a70f1\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " Sep 17 15:56:03.670295 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670216 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-envoy\") pod \"82906d2d-5c60-4e02-829f-d010c89a70f1\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " Sep 17 15:56:03.670295 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670292 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-credential-socket\") pod \"82906d2d-5c60-4e02-829f-d010c89a70f1\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " Sep 17 15:56:03.670471 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670330 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-workload-certs\") pod \"82906d2d-5c60-4e02-829f-d010c89a70f1\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " Sep 17 15:56:03.670471 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670349 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws9g4\" (UniqueName: \"kubernetes.io/projected/82906d2d-5c60-4e02-829f-d010c89a70f1-kube-api-access-ws9g4\") pod \"82906d2d-5c60-4e02-829f-d010c89a70f1\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " Sep 17 15:56:03.670471 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670370 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-podinfo\") pod \"82906d2d-5c60-4e02-829f-d010c89a70f1\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " Sep 17 15:56:03.670471 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670403 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-workload-socket\") pod \"82906d2d-5c60-4e02-829f-d010c89a70f1\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " Sep 17 15:56:03.670471 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670429 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82906d2d-5c60-4e02-829f-d010c89a70f1-istiod-ca-cert\") pod \"82906d2d-5c60-4e02-829f-d010c89a70f1\" (UID: \"82906d2d-5c60-4e02-829f-d010c89a70f1\") " Sep 17 15:56:03.670735 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670560 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-data" (OuterVolumeSpecName: "istio-data") pod "82906d2d-5c60-4e02-829f-d010c89a70f1" (UID: "82906d2d-5c60-4e02-829f-d010c89a70f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:56:03.670735 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670645 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82906d2d-5c60-4e02-829f-d010c89a70f1" (UID: "82906d2d-5c60-4e02-829f-d010c89a70f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:56:03.670839 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670768 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-credential-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:03.670839 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670789 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-data\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:03.670974 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670908 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82906d2d-5c60-4e02-829f-d010c89a70f1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82906d2d-5c60-4e02-829f-d010c89a70f1" (UID: "82906d2d-5c60-4e02-829f-d010c89a70f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:56:03.671057 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.670948 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82906d2d-5c60-4e02-829f-d010c89a70f1" (UID: "82906d2d-5c60-4e02-829f-d010c89a70f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:56:03.671116 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.671041 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82906d2d-5c60-4e02-829f-d010c89a70f1" (UID: "82906d2d-5c60-4e02-829f-d010c89a70f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:56:03.672555 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.672527 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82906d2d-5c60-4e02-829f-d010c89a70f1" (UID: "82906d2d-5c60-4e02-829f-d010c89a70f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:56:03.672838 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.672810 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82906d2d-5c60-4e02-829f-d010c89a70f1" (UID: "82906d2d-5c60-4e02-829f-d010c89a70f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 15:56:03.672838 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.672825 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "82906d2d-5c60-4e02-829f-d010c89a70f1" (UID: "82906d2d-5c60-4e02-829f-d010c89a70f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:56:03.673007 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.672938 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82906d2d-5c60-4e02-829f-d010c89a70f1-kube-api-access-ws9g4" (OuterVolumeSpecName: "kube-api-access-ws9g4") pod "82906d2d-5c60-4e02-829f-d010c89a70f1" (UID: "82906d2d-5c60-4e02-829f-d010c89a70f1"). InnerVolumeSpecName "kube-api-access-ws9g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:56:03.772171 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.772140 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-workload-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:03.772171 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.772166 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82906d2d-5c60-4e02-829f-d010c89a70f1-istiod-ca-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:03.772322 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.772179 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-token\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:03.772322 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.772190 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-envoy\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:03.772322 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.772202 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82906d2d-5c60-4e02-829f-d010c89a70f1-workload-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:03.772322 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.772213 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws9g4\" (UniqueName: \"kubernetes.io/projected/82906d2d-5c60-4e02-829f-d010c89a70f1-kube-api-access-ws9g4\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:03.772322 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.772228 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82906d2d-5c60-4e02-829f-d010c89a70f1-istio-podinfo\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:03.979981 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.979868 2581 generic.go:356] "Generic (PLEG): container finished" podID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerID="d45d4cc4bb078fe7ddea5da54407f1460f8d0b95e30607c32f092d460285eef9" exitCode=0 Sep 17 15:56:03.979981 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.979945 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" Sep 17 15:56:03.979981 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.979961 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" event={"ID":"82906d2d-5c60-4e02-829f-d010c89a70f1","Type":"ContainerDied","Data":"d45d4cc4bb078fe7ddea5da54407f1460f8d0b95e30607c32f092d460285eef9"} Sep 17 15:56:03.980250 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.980019 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw" event={"ID":"82906d2d-5c60-4e02-829f-d010c89a70f1","Type":"ContainerDied","Data":"a1c2e7993ed5e44802626a66affb7853b2526fe7f28c72b411dc18dccbcfaf96"} Sep 17 15:56:03.980250 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.980050 2581 scope.go:117] "RemoveContainer" containerID="d45d4cc4bb078fe7ddea5da54407f1460f8d0b95e30607c32f092d460285eef9" Sep 17 15:56:03.990234 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.990216 2581 scope.go:117] "RemoveContainer" containerID="fc9d5068c27ab4d61bc489df8dfe0cdc53c99a4ca97f66d90ed22f0473093414" Sep 17 15:56:03.998708 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.998693 2581 scope.go:117] "RemoveContainer" containerID="d45d4cc4bb078fe7ddea5da54407f1460f8d0b95e30607c32f092d460285eef9" Sep 17 15:56:03.999009 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:56:03.998983 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45d4cc4bb078fe7ddea5da54407f1460f8d0b95e30607c32f092d460285eef9\": container with ID starting with d45d4cc4bb078fe7ddea5da54407f1460f8d0b95e30607c32f092d460285eef9 not found: ID does not exist" containerID="d45d4cc4bb078fe7ddea5da54407f1460f8d0b95e30607c32f092d460285eef9" Sep 17 15:56:03.999081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.999022 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45d4cc4bb078fe7ddea5da54407f1460f8d0b95e30607c32f092d460285eef9"} err="failed to get container status \"d45d4cc4bb078fe7ddea5da54407f1460f8d0b95e30607c32f092d460285eef9\": rpc error: code = NotFound desc = could not find container \"d45d4cc4bb078fe7ddea5da54407f1460f8d0b95e30607c32f092d460285eef9\": container with ID starting with d45d4cc4bb078fe7ddea5da54407f1460f8d0b95e30607c32f092d460285eef9 not found: ID does not exist" Sep 17 15:56:03.999081 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.999046 2581 scope.go:117] "RemoveContainer" containerID="fc9d5068c27ab4d61bc489df8dfe0cdc53c99a4ca97f66d90ed22f0473093414" Sep 17 15:56:03.999314 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:56:03.999297 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9d5068c27ab4d61bc489df8dfe0cdc53c99a4ca97f66d90ed22f0473093414\": container with ID starting with fc9d5068c27ab4d61bc489df8dfe0cdc53c99a4ca97f66d90ed22f0473093414 not found: ID does not exist" containerID="fc9d5068c27ab4d61bc489df8dfe0cdc53c99a4ca97f66d90ed22f0473093414" Sep 17 15:56:03.999360 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:03.999320 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9d5068c27ab4d61bc489df8dfe0cdc53c99a4ca97f66d90ed22f0473093414"} err="failed to get container status \"fc9d5068c27ab4d61bc489df8dfe0cdc53c99a4ca97f66d90ed22f0473093414\": rpc error: code = NotFound desc = could not find container \"fc9d5068c27ab4d61bc489df8dfe0cdc53c99a4ca97f66d90ed22f0473093414\": container with ID starting with fc9d5068c27ab4d61bc489df8dfe0cdc53c99a4ca97f66d90ed22f0473093414 not found: ID does not exist" Sep 17 15:56:04.009807 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:04.009782 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw"] Sep 17 15:56:04.018798 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:04.018774 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c686d0-openshift-default-79744f8746-j49hw"] Sep 17 15:56:04.056320 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:04.056298 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:04.160567 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:04.160520 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:04.936144 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:04.936115 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:05.462247 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:05.462208 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82906d2d-5c60-4e02-829f-d010c89a70f1" path="/var/lib/kubelet/pods/82906d2d-5c60-4e02-829f-d010c89a70f1/volumes" Sep 17 15:56:06.108071 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:06.108038 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:08.905281 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:08.905252 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:14.704449 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:14.704391 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:14.762217 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:14.762165 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt"] Sep 17 15:56:14.762522 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:14.762494 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" podUID="f4266c95-b4a2-4760-bdb3-ebdb38db785a" containerName="payload-processing" containerID="cri-o://e013bfb95fc16402c7e094f6106b688734e0d42e33a1751aa49c9f6a72defe16" gracePeriod=30 Sep 17 15:56:14.798727 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:14.798696 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67"] Sep 17 15:56:14.798996 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:14.798971 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" podUID="f75059c4-89b5-42f4-afba-4b454cf37500" containerName="payload-pre-processing" containerID="cri-o://1f253fbdcfd9555618dca1f424afc323deb4b70a14b2d97881d51864fcce2d4f" gracePeriod=30 Sep 17 15:56:14.871540 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:14.871500 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:15.047864 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.047835 2581 generic.go:356] "Generic (PLEG): container finished" podID="f75059c4-89b5-42f4-afba-4b454cf37500" containerID="1f253fbdcfd9555618dca1f424afc323deb4b70a14b2d97881d51864fcce2d4f" exitCode=0 Sep 17 15:56:15.048024 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.047908 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" event={"ID":"f75059c4-89b5-42f4-afba-4b454cf37500","Type":"ContainerDied","Data":"1f253fbdcfd9555618dca1f424afc323deb4b70a14b2d97881d51864fcce2d4f"} Sep 17 15:56:15.049455 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.049428 2581 generic.go:356] "Generic (PLEG): container finished" podID="f4266c95-b4a2-4760-bdb3-ebdb38db785a" containerID="e013bfb95fc16402c7e094f6106b688734e0d42e33a1751aa49c9f6a72defe16" exitCode=0 Sep 17 15:56:15.049579 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.049501 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" event={"ID":"f4266c95-b4a2-4760-bdb3-ebdb38db785a","Type":"ContainerDied","Data":"e013bfb95fc16402c7e094f6106b688734e0d42e33a1751aa49c9f6a72defe16"} Sep 17 15:56:15.068483 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.068465 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" Sep 17 15:56:15.071291 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.071274 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" Sep 17 15:56:15.083237 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.083217 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:15.166966 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.166932 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75059c4-89b5-42f4-afba-4b454cf37500-plugins-config-volume\") pod \"f75059c4-89b5-42f4-afba-4b454cf37500\" (UID: \"f75059c4-89b5-42f4-afba-4b454cf37500\") " Sep 17 15:56:15.167108 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.166977 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwh7g\" (UniqueName: \"kubernetes.io/projected/f4266c95-b4a2-4760-bdb3-ebdb38db785a-kube-api-access-mwh7g\") pod \"f4266c95-b4a2-4760-bdb3-ebdb38db785a\" (UID: \"f4266c95-b4a2-4760-bdb3-ebdb38db785a\") " Sep 17 15:56:15.167108 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.167003 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c2mm\" (UniqueName: \"kubernetes.io/projected/f75059c4-89b5-42f4-afba-4b454cf37500-kube-api-access-2c2mm\") pod \"f75059c4-89b5-42f4-afba-4b454cf37500\" (UID: \"f75059c4-89b5-42f4-afba-4b454cf37500\") " Sep 17 15:56:15.167108 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.167035 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4266c95-b4a2-4760-bdb3-ebdb38db785a-plugins-config-volume\") pod \"f4266c95-b4a2-4760-bdb3-ebdb38db785a\" (UID: \"f4266c95-b4a2-4760-bdb3-ebdb38db785a\") " Sep 17 15:56:15.167435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.167414 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f75059c4-89b5-42f4-afba-4b454cf37500-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f75059c4-89b5-42f4-afba-4b454cf37500" (UID: "f75059c4-89b5-42f4-afba-4b454cf37500"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:56:15.167518 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.167473 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4266c95-b4a2-4760-bdb3-ebdb38db785a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4266c95-b4a2-4760-bdb3-ebdb38db785a" (UID: "f4266c95-b4a2-4760-bdb3-ebdb38db785a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:56:15.169619 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.169594 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4266c95-b4a2-4760-bdb3-ebdb38db785a-kube-api-access-mwh7g" (OuterVolumeSpecName: "kube-api-access-mwh7g") pod "f4266c95-b4a2-4760-bdb3-ebdb38db785a" (UID: "f4266c95-b4a2-4760-bdb3-ebdb38db785a"). InnerVolumeSpecName "kube-api-access-mwh7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:56:15.169859 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.169837 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75059c4-89b5-42f4-afba-4b454cf37500-kube-api-access-2c2mm" (OuterVolumeSpecName: "kube-api-access-2c2mm") pod "f75059c4-89b5-42f4-afba-4b454cf37500" (UID: "f75059c4-89b5-42f4-afba-4b454cf37500"). InnerVolumeSpecName "kube-api-access-2c2mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:56:15.174872 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.174838 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb"] Sep 17 15:56:15.179237 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.179200 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f99a1-8nmkb"] Sep 17 15:56:15.268323 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.268244 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75059c4-89b5-42f4-afba-4b454cf37500-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:15.268323 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.268273 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwh7g\" (UniqueName: \"kubernetes.io/projected/f4266c95-b4a2-4760-bdb3-ebdb38db785a-kube-api-access-mwh7g\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:15.268323 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.268284 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c2mm\" (UniqueName: \"kubernetes.io/projected/f75059c4-89b5-42f4-afba-4b454cf37500-kube-api-access-2c2mm\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:15.268323 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.268294 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4266c95-b4a2-4760-bdb3-ebdb38db785a-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:15.324020 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.323988 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:15.461957 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.461903 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67776141-3c2d-4b85-bbda-b6800ad3dba8" path="/var/lib/kubelet/pods/67776141-3c2d-4b85-bbda-b6800ad3dba8/volumes" Sep 17 15:56:15.566476 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.566446 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:15.674337 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:15.674309 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:16.055884 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:16.055852 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" Sep 17 15:56:16.056349 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:16.055901 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt" event={"ID":"f4266c95-b4a2-4760-bdb3-ebdb38db785a","Type":"ContainerDied","Data":"a204814f61db1f063e8188d45313587cda7dd84c68c7545920aca1bf4c2d9ae5"} Sep 17 15:56:16.056349 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:16.055980 2581 scope.go:117] "RemoveContainer" containerID="e013bfb95fc16402c7e094f6106b688734e0d42e33a1751aa49c9f6a72defe16" Sep 17 15:56:16.057377 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:16.057331 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" event={"ID":"f75059c4-89b5-42f4-afba-4b454cf37500","Type":"ContainerDied","Data":"8f4c6f2617c549c94f5c425198e1535debe91566481d0182e019b12a3f79046b"} Sep 17 15:56:16.057377 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:16.057360 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67" Sep 17 15:56:16.065155 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:16.065139 2581 scope.go:117] "RemoveContainer" containerID="1f253fbdcfd9555618dca1f424afc323deb4b70a14b2d97881d51864fcce2d4f" Sep 17 15:56:16.087127 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:16.087106 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt"] Sep 17 15:56:16.097669 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:16.097645 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8f99a1-ddc447dd5-8cplt"] Sep 17 15:56:16.121763 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:16.121735 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67"] Sep 17 15:56:16.125871 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:16.125851 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8f99a1-5d9d85dfc8-kpl67"] Sep 17 15:56:16.460289 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:16.460203 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:17.462615 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:17.462583 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4266c95-b4a2-4760-bdb3-ebdb38db785a" path="/var/lib/kubelet/pods/f4266c95-b4a2-4760-bdb3-ebdb38db785a/volumes" Sep 17 15:56:17.463007 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:17.462935 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f75059c4-89b5-42f4-afba-4b454cf37500" path="/var/lib/kubelet/pods/f75059c4-89b5-42f4-afba-4b454cf37500/volumes" Sep 17 15:56:17.502422 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:17.502398 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:19.114544 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:19.114483 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:20.729619 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:20.729584 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:24.693461 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:24.693428 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:24.864015 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:24.863974 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:24.993459 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:24.993347 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:25.274514 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.274431 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh"] Sep 17 15:56:25.274892 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.274874 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4266c95-b4a2-4760-bdb3-ebdb38db785a" containerName="payload-processing" Sep 17 15:56:25.275024 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.274896 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4266c95-b4a2-4760-bdb3-ebdb38db785a" containerName="payload-processing" Sep 17 15:56:25.275024 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.274970 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerName="istio-proxy" Sep 17 15:56:25.275024 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.274983 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerName="istio-proxy" Sep 17 15:56:25.275024 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.274996 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67776141-3c2d-4b85-bbda-b6800ad3dba8" containerName="revoke-keys" Sep 17 15:56:25.275024 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.275005 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="67776141-3c2d-4b85-bbda-b6800ad3dba8" containerName="revoke-keys" Sep 17 15:56:25.275024 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.275024 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f75059c4-89b5-42f4-afba-4b454cf37500" containerName="payload-pre-processing" Sep 17 15:56:25.275327 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.275033 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f75059c4-89b5-42f4-afba-4b454cf37500" containerName="payload-pre-processing" Sep 17 15:56:25.275327 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.275147 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f75059c4-89b5-42f4-afba-4b454cf37500" containerName="payload-pre-processing" Sep 17 15:56:25.275327 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.275161 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerName="istio-proxy" Sep 17 15:56:25.275327 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.275173 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4266c95-b4a2-4760-bdb3-ebdb38db785a" containerName="payload-processing" Sep 17 15:56:25.275327 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.275183 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="67776141-3c2d-4b85-bbda-b6800ad3dba8" containerName="revoke-keys" Sep 17 15:56:25.279743 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.279724 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh" Sep 17 15:56:25.282838 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.282820 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\"" Sep 17 15:56:25.292755 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.292734 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 15:56:25.300235 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.300214 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh"] Sep 17 15:56:25.353694 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.353663 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f583e06a-376f-4957-a898-da6b53e77aa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh\" (UID: \"f583e06a-376f-4957-a898-da6b53e77aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh" Sep 17 15:56:25.455031 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.454999 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f583e06a-376f-4957-a898-da6b53e77aa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh\" (UID: \"f583e06a-376f-4957-a898-da6b53e77aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh" Sep 17 15:56:25.455583 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.455565 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f583e06a-376f-4957-a898-da6b53e77aa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh\" (UID: \"f583e06a-376f-4957-a898-da6b53e77aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh" Sep 17 15:56:25.590184 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.590157 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh" Sep 17 15:56:25.682614 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.682577 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:25.912338 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:25.912312 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh"] Sep 17 15:56:25.913663 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:56:25.913639 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf583e06a_376f_4957_a898_da6b53e77aa8.slice/crio-39e2a228f4b3627b4f26d089351d619e6d9389a448f242ed80e6c0546b995684 WatchSource:0}: Error finding container 39e2a228f4b3627b4f26d089351d619e6d9389a448f242ed80e6c0546b995684: Status 404 returned error can't find the container with id 39e2a228f4b3627b4f26d089351d619e6d9389a448f242ed80e6c0546b995684 Sep 17 15:56:26.115497 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:26.115464 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh" event={"ID":"f583e06a-376f-4957-a898-da6b53e77aa8","Type":"ContainerStarted","Data":"54202b1d5abba636d0960452c0869db80b376aa6f02679efc58e05d9a1a1b77e"} Sep 17 15:56:26.115613 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:26.115532 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh" event={"ID":"f583e06a-376f-4957-a898-da6b53e77aa8","Type":"ContainerStarted","Data":"39e2a228f4b3627b4f26d089351d619e6d9389a448f242ed80e6c0546b995684"} Sep 17 15:56:26.153411 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:26.153364 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh" podStartSLOduration=1.153347621 podStartE2EDuration="1.153347621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:26.151653137 +0000 UTC m=+1573.147889882" watchObservedRunningTime="2026-09-17 15:56:26.153347621 +0000 UTC m=+1573.149584364" Sep 17 15:56:26.996741 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:26.996708 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:27.122091 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:27.122054 2581 generic.go:356] "Generic (PLEG): container finished" podID="f583e06a-376f-4957-a898-da6b53e77aa8" containerID="54202b1d5abba636d0960452c0869db80b376aa6f02679efc58e05d9a1a1b77e" exitCode=0 Sep 17 15:56:27.122254 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:27.122118 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh" event={"ID":"f583e06a-376f-4957-a898-da6b53e77aa8","Type":"ContainerDied","Data":"54202b1d5abba636d0960452c0869db80b376aa6f02679efc58e05d9a1a1b77e"} Sep 17 15:56:27.803484 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:27.803445 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:28.257149 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:28.257128 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh" Sep 17 15:56:28.382775 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:28.382688 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f583e06a-376f-4957-a898-da6b53e77aa8-maas-api-service-ca\") pod \"f583e06a-376f-4957-a898-da6b53e77aa8\" (UID: \"f583e06a-376f-4957-a898-da6b53e77aa8\") " Sep 17 15:56:28.383078 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:28.383056 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f583e06a-376f-4957-a898-da6b53e77aa8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f583e06a-376f-4957-a898-da6b53e77aa8" (UID: "f583e06a-376f-4957-a898-da6b53e77aa8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:56:28.484001 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:28.483969 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f583e06a-376f-4957-a898-da6b53e77aa8-maas-api-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:28.693294 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:28.693213 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:29.135135 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:29.135106 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh" Sep 17 15:56:29.135135 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:29.135129 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh" event={"ID":"f583e06a-376f-4957-a898-da6b53e77aa8","Type":"ContainerDied","Data":"39e2a228f4b3627b4f26d089351d619e6d9389a448f242ed80e6c0546b995684"} Sep 17 15:56:29.135349 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:29.135197 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39e2a228f4b3627b4f26d089351d619e6d9389a448f242ed80e6c0546b995684" Sep 17 15:56:29.426148 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:29.426063 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:30.182394 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:30.182364 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:36.155825 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.155794 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:36.233431 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.233400 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr"] Sep 17 15:56:36.233658 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.233633 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" podUID="6ff6d1d6-ca57-43d7-a213-4451821f437a" containerName="payload-pre-processing" containerID="cri-o://a60c6f82f751532403ddd18521070d6161354850a95fab95ecdbf1d1e5c2674a" gracePeriod=30 Sep 17 15:56:36.355448 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.355398 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:36.460212 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.460130 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:36.476962 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.476931 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh"] Sep 17 15:56:36.482967 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.482940 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2b23c7-wl8hh"] Sep 17 15:56:36.495398 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.495369 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" Sep 17 15:56:36.552009 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.551978 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:36.556314 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.556291 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg8sw\" (UniqueName: \"kubernetes.io/projected/6ff6d1d6-ca57-43d7-a213-4451821f437a-kube-api-access-vg8sw\") pod \"6ff6d1d6-ca57-43d7-a213-4451821f437a\" (UID: \"6ff6d1d6-ca57-43d7-a213-4451821f437a\") " Sep 17 15:56:36.556436 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.556355 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff6d1d6-ca57-43d7-a213-4451821f437a-plugins-config-volume\") pod \"6ff6d1d6-ca57-43d7-a213-4451821f437a\" (UID: \"6ff6d1d6-ca57-43d7-a213-4451821f437a\") " Sep 17 15:56:36.556757 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.556733 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff6d1d6-ca57-43d7-a213-4451821f437a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ff6d1d6-ca57-43d7-a213-4451821f437a" (UID: "6ff6d1d6-ca57-43d7-a213-4451821f437a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:56:36.558426 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.558402 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff6d1d6-ca57-43d7-a213-4451821f437a-kube-api-access-vg8sw" (OuterVolumeSpecName: "kube-api-access-vg8sw") pod "6ff6d1d6-ca57-43d7-a213-4451821f437a" (UID: "6ff6d1d6-ca57-43d7-a213-4451821f437a"). InnerVolumeSpecName "kube-api-access-vg8sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:56:36.657689 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.657658 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff6d1d6-ca57-43d7-a213-4451821f437a-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:36.657689 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:36.657686 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg8sw\" (UniqueName: \"kubernetes.io/projected/6ff6d1d6-ca57-43d7-a213-4451821f437a-kube-api-access-vg8sw\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:37.131515 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.131479 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:37.177694 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.177661 2581 generic.go:356] "Generic (PLEG): container finished" podID="6ff6d1d6-ca57-43d7-a213-4451821f437a" containerID="a60c6f82f751532403ddd18521070d6161354850a95fab95ecdbf1d1e5c2674a" exitCode=0 Sep 17 15:56:37.178086 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.177705 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" Sep 17 15:56:37.178086 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.177713 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" event={"ID":"6ff6d1d6-ca57-43d7-a213-4451821f437a","Type":"ContainerDied","Data":"a60c6f82f751532403ddd18521070d6161354850a95fab95ecdbf1d1e5c2674a"} Sep 17 15:56:37.178086 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.177756 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr" event={"ID":"6ff6d1d6-ca57-43d7-a213-4451821f437a","Type":"ContainerDied","Data":"71d5df7ce98958db639650b68268a1fb436df815539e99e39edf2b2f89d02fbc"} Sep 17 15:56:37.178086 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.177777 2581 scope.go:117] "RemoveContainer" containerID="a60c6f82f751532403ddd18521070d6161354850a95fab95ecdbf1d1e5c2674a" Sep 17 15:56:37.187504 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.187488 2581 scope.go:117] "RemoveContainer" containerID="a60c6f82f751532403ddd18521070d6161354850a95fab95ecdbf1d1e5c2674a" Sep 17 15:56:37.187781 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:56:37.187762 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60c6f82f751532403ddd18521070d6161354850a95fab95ecdbf1d1e5c2674a\": container with ID starting with a60c6f82f751532403ddd18521070d6161354850a95fab95ecdbf1d1e5c2674a not found: ID does not exist" containerID="a60c6f82f751532403ddd18521070d6161354850a95fab95ecdbf1d1e5c2674a" Sep 17 15:56:37.187833 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.187790 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60c6f82f751532403ddd18521070d6161354850a95fab95ecdbf1d1e5c2674a"} err="failed to get container status \"a60c6f82f751532403ddd18521070d6161354850a95fab95ecdbf1d1e5c2674a\": rpc error: code = NotFound desc = could not find container \"a60c6f82f751532403ddd18521070d6161354850a95fab95ecdbf1d1e5c2674a\": container with ID starting with a60c6f82f751532403ddd18521070d6161354850a95fab95ecdbf1d1e5c2674a not found: ID does not exist" Sep 17 15:56:37.209273 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.209253 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr"] Sep 17 15:56:37.213231 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.213210 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2b23c7-794bd6cbdb-x87nr"] Sep 17 15:56:37.258381 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.258355 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:37.462204 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.462112 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ff6d1d6-ca57-43d7-a213-4451821f437a" path="/var/lib/kubelet/pods/6ff6d1d6-ca57-43d7-a213-4451821f437a/volumes" Sep 17 15:56:37.462565 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.462544 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f583e06a-376f-4957-a898-da6b53e77aa8" path="/var/lib/kubelet/pods/f583e06a-376f-4957-a898-da6b53e77aa8/volumes" Sep 17 15:56:37.565536 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.565498 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7791-0e7cb2"] Sep 17 15:56:37.565899 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.565881 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f583e06a-376f-4957-a898-da6b53e77aa8" containerName="revoke-keys" Sep 17 15:56:37.566018 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.565901 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f583e06a-376f-4957-a898-da6b53e77aa8" containerName="revoke-keys" Sep 17 15:56:37.566018 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.565938 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ff6d1d6-ca57-43d7-a213-4451821f437a" containerName="payload-pre-processing" Sep 17 15:56:37.566018 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.565947 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ff6d1d6-ca57-43d7-a213-4451821f437a" containerName="payload-pre-processing" Sep 17 15:56:37.566018 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.565961 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerName="istio-proxy" Sep 17 15:56:37.566018 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.565969 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerName="istio-proxy" Sep 17 15:56:37.566274 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.566081 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ff6d1d6-ca57-43d7-a213-4451821f437a" containerName="payload-pre-processing" Sep 17 15:56:37.566274 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.566094 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="82906d2d-5c60-4e02-829f-d010c89a70f1" containerName="istio-proxy" Sep 17 15:56:37.566274 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.566105 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f583e06a-376f-4957-a898-da6b53e77aa8" containerName="revoke-keys" Sep 17 15:56:37.570273 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.570253 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7791-0e7cb2" Sep 17 15:56:37.574115 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.574084 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-55clv\"" Sep 17 15:56:37.577180 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.577155 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7791-0e7cb2"] Sep 17 15:56:37.666587 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.666553 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9cnz\" (UniqueName: \"kubernetes.io/projected/17603aaa-0180-4815-94b7-3da01a45ee15-kube-api-access-c9cnz\") pod \"test-curl-7791-0e7cb2\" (UID: \"17603aaa-0180-4815-94b7-3da01a45ee15\") " pod="opendatahub/test-curl-7791-0e7cb2" Sep 17 15:56:37.767764 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.767684 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cnz\" (UniqueName: \"kubernetes.io/projected/17603aaa-0180-4815-94b7-3da01a45ee15-kube-api-access-c9cnz\") pod \"test-curl-7791-0e7cb2\" (UID: \"17603aaa-0180-4815-94b7-3da01a45ee15\") " pod="opendatahub/test-curl-7791-0e7cb2" Sep 17 15:56:37.779923 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.779884 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9cnz\" (UniqueName: \"kubernetes.io/projected/17603aaa-0180-4815-94b7-3da01a45ee15-kube-api-access-c9cnz\") pod \"test-curl-7791-0e7cb2\" (UID: \"17603aaa-0180-4815-94b7-3da01a45ee15\") " pod="opendatahub/test-curl-7791-0e7cb2" Sep 17 15:56:37.880596 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:37.880568 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7791-0e7cb2" Sep 17 15:56:38.159846 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:38.159812 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7791-0e7cb2"] Sep 17 15:56:38.171639 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:56:38.171609 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17603aaa_0180_4815_94b7_3da01a45ee15.slice/crio-d2e3aa3d859889e26f23f59d450d8734c123f536605718a8cbde291c291f81f5 WatchSource:0}: Error finding container d2e3aa3d859889e26f23f59d450d8734c123f536605718a8cbde291c291f81f5: Status 404 returned error can't find the container with id d2e3aa3d859889e26f23f59d450d8734c123f536605718a8cbde291c291f81f5 Sep 17 15:56:38.183537 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:38.183504 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7791-0e7cb2" event={"ID":"17603aaa-0180-4815-94b7-3da01a45ee15","Type":"ContainerStarted","Data":"d2e3aa3d859889e26f23f59d450d8734c123f536605718a8cbde291c291f81f5"} Sep 17 15:56:38.313362 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:38.313331 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:39.191449 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:39.191408 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7791-0e7cb2" event={"ID":"17603aaa-0180-4815-94b7-3da01a45ee15","Type":"ContainerStarted","Data":"d156abd5ebdec6edde34cbf46d118fdf5ccf278f219976e1cf06997a301d87fb"} Sep 17 15:56:39.384928 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:39.384881 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:40.034056 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:40.034004 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7791-0e7cb2" podStartSLOduration=3.033987238 podStartE2EDuration="3.033987238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:39.211045325 +0000 UTC m=+1586.207282071" watchObservedRunningTime="2026-09-17 15:56:40.033987238 +0000 UTC m=+1587.030223984" Sep 17 15:56:40.035454 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:40.035425 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7791-0e7cb2"] Sep 17 15:56:40.042563 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:40.042536 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7791-0e7cb2"] Sep 17 15:56:40.196615 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:56:40.196587 2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7791-0e7cb2" container="test-curl-7791-0e7cb2" Sep 17 15:56:40.302790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:40.302706 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:41.201783 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:41.201739 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7791-0e7cb2" podUID="17603aaa-0180-4815-94b7-3da01a45ee15" containerName="test-curl-7791-0e7cb2" containerID="cri-o://d156abd5ebdec6edde34cbf46d118fdf5ccf278f219976e1cf06997a301d87fb" gracePeriod=2 Sep 17 15:56:41.204831 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:56:41.204791 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7791-0e7cb2\" is forbidden: User \"system:node:ip-10-0-128-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-15.ec2.internal' and this object" podUID="17603aaa-0180-4815-94b7-3da01a45ee15" pod="opendatahub/test-curl-7791-0e7cb2" Sep 17 15:56:41.519422 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:41.519334 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws"] Sep 17 15:56:41.553026 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:41.552986 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:42.624741 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:42.624711 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:43.383475 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:43.383450 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7791-0e7cb2" Sep 17 15:56:43.417701 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:43.417676 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9cnz\" (UniqueName: \"kubernetes.io/projected/17603aaa-0180-4815-94b7-3da01a45ee15-kube-api-access-c9cnz\") pod \"17603aaa-0180-4815-94b7-3da01a45ee15\" (UID: \"17603aaa-0180-4815-94b7-3da01a45ee15\") " Sep 17 15:56:43.419662 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:43.419642 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17603aaa-0180-4815-94b7-3da01a45ee15-kube-api-access-c9cnz" (OuterVolumeSpecName: "kube-api-access-c9cnz") pod "17603aaa-0180-4815-94b7-3da01a45ee15" (UID: "17603aaa-0180-4815-94b7-3da01a45ee15"). InnerVolumeSpecName "kube-api-access-c9cnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:56:43.461435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:43.461358 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17603aaa-0180-4815-94b7-3da01a45ee15" path="/var/lib/kubelet/pods/17603aaa-0180-4815-94b7-3da01a45ee15/volumes" Sep 17 15:56:43.518569 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:43.518538 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9cnz\" (UniqueName: \"kubernetes.io/projected/17603aaa-0180-4815-94b7-3da01a45ee15-kube-api-access-c9cnz\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:44.219446 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:44.219411 2581 generic.go:356] "Generic (PLEG): container finished" podID="17603aaa-0180-4815-94b7-3da01a45ee15" containerID="d156abd5ebdec6edde34cbf46d118fdf5ccf278f219976e1cf06997a301d87fb" exitCode=137 Sep 17 15:56:44.219931 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:44.219463 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7791-0e7cb2" Sep 17 15:56:44.219931 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:44.219473 2581 scope.go:117] "RemoveContainer" containerID="d156abd5ebdec6edde34cbf46d118fdf5ccf278f219976e1cf06997a301d87fb" Sep 17 15:56:44.231062 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:44.231039 2581 scope.go:117] "RemoveContainer" containerID="d156abd5ebdec6edde34cbf46d118fdf5ccf278f219976e1cf06997a301d87fb" Sep 17 15:56:44.231553 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:56:44.231528 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d156abd5ebdec6edde34cbf46d118fdf5ccf278f219976e1cf06997a301d87fb\": container with ID starting with d156abd5ebdec6edde34cbf46d118fdf5ccf278f219976e1cf06997a301d87fb not found: ID does not exist" containerID="d156abd5ebdec6edde34cbf46d118fdf5ccf278f219976e1cf06997a301d87fb" Sep 17 15:56:44.231600 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:44.231561 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d156abd5ebdec6edde34cbf46d118fdf5ccf278f219976e1cf06997a301d87fb"} err="failed to get container status \"d156abd5ebdec6edde34cbf46d118fdf5ccf278f219976e1cf06997a301d87fb\": rpc error: code = NotFound desc = could not find container \"d156abd5ebdec6edde34cbf46d118fdf5ccf278f219976e1cf06997a301d87fb\": container with ID starting with d156abd5ebdec6edde34cbf46d118fdf5ccf278f219976e1cf06997a301d87fb not found: ID does not exist" Sep 17 15:56:44.324316 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:44.324282 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:44.475410 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:44.475334 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:44.763645 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:44.763571 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:45.036500 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.036469 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l"] Sep 17 15:56:45.036827 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.036815 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17603aaa-0180-4815-94b7-3da01a45ee15" containerName="test-curl-7791-0e7cb2" Sep 17 15:56:45.036873 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.036828 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="17603aaa-0180-4815-94b7-3da01a45ee15" containerName="test-curl-7791-0e7cb2" Sep 17 15:56:45.036935 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.036881 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="17603aaa-0180-4815-94b7-3da01a45ee15" containerName="test-curl-7791-0e7cb2" Sep 17 15:56:45.039781 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.039762 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l" Sep 17 15:56:45.043048 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.043028 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 15:56:45.043221 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.043203 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\"" Sep 17 15:56:45.052424 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.052401 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l"] Sep 17 15:56:45.133811 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.133773 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b60cad4-7821-4f13-9766-cb303d66bf76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l\" (UID: \"5b60cad4-7821-4f13-9766-cb303d66bf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l" Sep 17 15:56:45.234664 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.234625 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b60cad4-7821-4f13-9766-cb303d66bf76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l\" (UID: \"5b60cad4-7821-4f13-9766-cb303d66bf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l" Sep 17 15:56:45.235259 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.235242 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b60cad4-7821-4f13-9766-cb303d66bf76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l\" (UID: \"5b60cad4-7821-4f13-9766-cb303d66bf76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l" Sep 17 15:56:45.323713 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.323617 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:45.349757 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.349719 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l" Sep 17 15:56:45.667150 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:45.667104 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l"] Sep 17 15:56:45.667897 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:56:45.667870 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b60cad4_7821_4f13_9766_cb303d66bf76.slice/crio-11cc8e06c904a23c36641b40de273b4a4b6ebe719e715c3b4d09d855c1562043 WatchSource:0}: Error finding container 11cc8e06c904a23c36641b40de273b4a4b6ebe719e715c3b4d09d855c1562043: Status 404 returned error can't find the container with id 11cc8e06c904a23c36641b40de273b4a4b6ebe719e715c3b4d09d855c1562043 Sep 17 15:56:46.232267 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.232231 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l" event={"ID":"5b60cad4-7821-4f13-9766-cb303d66bf76","Type":"ContainerStarted","Data":"209f18e27a48bc617c509d789ec968ddccdfc234499737645909b7a6ba453470"} Sep 17 15:56:46.232442 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.232328 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l" event={"ID":"5b60cad4-7821-4f13-9766-cb303d66bf76","Type":"ContainerStarted","Data":"11cc8e06c904a23c36641b40de273b4a4b6ebe719e715c3b4d09d855c1562043"} Sep 17 15:56:46.263967 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.263901 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l" podStartSLOduration=1.263887587 podStartE2EDuration="1.263887587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:46.263228307 +0000 UTC m=+1593.259465055" watchObservedRunningTime="2026-09-17 15:56:46.263887587 +0000 UTC m=+1593.260124333" Sep 17 15:56:46.782139 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.782098 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq"] Sep 17 15:56:46.785466 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.785444 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.787906 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.787890 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d63c0\"/\"sub-test-model-1d63c0-kserve-self-signed-certs\"" Sep 17 15:56:46.804367 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.804346 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq"] Sep 17 15:56:46.812092 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.812071 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:46.849631 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.849604 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-home\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.849745 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.849639 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-tmp-dir\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.849745 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.849663 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a16355-2030-4db0-bab7-69805bf1ae66-tls-certs\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.849824 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.849771 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-dshm\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.849824 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.849807 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-model-cache\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.849905 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.849888 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x5z6\" (UniqueName: \"kubernetes.io/projected/96a16355-2030-4db0-bab7-69805bf1ae66-kube-api-access-6x5z6\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.951266 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.951229 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-dshm\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.951266 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.951270 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-model-cache\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.951436 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.951399 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x5z6\" (UniqueName: \"kubernetes.io/projected/96a16355-2030-4db0-bab7-69805bf1ae66-kube-api-access-6x5z6\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.951492 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.951447 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-home\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.951645 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.951623 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-tmp-dir\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.951700 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.951663 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a16355-2030-4db0-bab7-69805bf1ae66-tls-certs\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.951700 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.951633 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-model-cache\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.951796 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.951724 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-home\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.951967 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.951945 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-tmp-dir\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.953541 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.953519 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-dshm\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.954144 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.954125 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a16355-2030-4db0-bab7-69805bf1ae66-tls-certs\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:46.961395 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:46.961370 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x5z6\" (UniqueName: \"kubernetes.io/projected/96a16355-2030-4db0-bab7-69805bf1ae66-kube-api-access-6x5z6\") pod \"sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:47.094533 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:47.094459 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:47.124009 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:47.123982 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:47.239509 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:47.239471 2581 generic.go:356] "Generic (PLEG): container finished" podID="5b60cad4-7821-4f13-9766-cb303d66bf76" containerID="209f18e27a48bc617c509d789ec968ddccdfc234499737645909b7a6ba453470" exitCode=0 Sep 17 15:56:47.239660 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:47.239537 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l" event={"ID":"5b60cad4-7821-4f13-9766-cb303d66bf76","Type":"ContainerDied","Data":"209f18e27a48bc617c509d789ec968ddccdfc234499737645909b7a6ba453470"} Sep 17 15:56:47.385279 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:47.385254 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq"] Sep 17 15:56:47.387073 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:56:47.387044 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96a16355_2030_4db0_bab7_69805bf1ae66.slice/crio-eb8cf132ad67a3b0bdfdada46555da0144454568112bb21f64fa13f56de8f1a8 WatchSource:0}: Error finding container eb8cf132ad67a3b0bdfdada46555da0144454568112bb21f64fa13f56de8f1a8: Status 404 returned error can't find the container with id eb8cf132ad67a3b0bdfdada46555da0144454568112bb21f64fa13f56de8f1a8 Sep 17 15:56:48.246723 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:48.246688 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" event={"ID":"96a16355-2030-4db0-bab7-69805bf1ae66","Type":"ContainerStarted","Data":"14471cbf566687ad44e84e1c04cdbe22217fa33cdf65f67ad7a957c0e2f3fe2a"} Sep 17 15:56:48.246904 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:48.246730 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" event={"ID":"96a16355-2030-4db0-bab7-69805bf1ae66","Type":"ContainerStarted","Data":"eb8cf132ad67a3b0bdfdada46555da0144454568112bb21f64fa13f56de8f1a8"} Sep 17 15:56:48.288645 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:48.288592 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" podStartSLOduration=2.288574094 podStartE2EDuration="2.288574094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:48.28659441 +0000 UTC m=+1595.282831178" watchObservedRunningTime="2026-09-17 15:56:48.288574094 +0000 UTC m=+1595.284810839" Sep 17 15:56:48.383941 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:48.383904 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l" Sep 17 15:56:48.464217 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:48.464186 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b60cad4-7821-4f13-9766-cb303d66bf76-maas-api-service-ca\") pod \"5b60cad4-7821-4f13-9766-cb303d66bf76\" (UID: \"5b60cad4-7821-4f13-9766-cb303d66bf76\") " Sep 17 15:56:48.464629 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:48.464502 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b60cad4-7821-4f13-9766-cb303d66bf76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b60cad4-7821-4f13-9766-cb303d66bf76" (UID: "5b60cad4-7821-4f13-9766-cb303d66bf76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:56:48.564943 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:48.564876 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b60cad4-7821-4f13-9766-cb303d66bf76-maas-api-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:48.613051 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:48.613022 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:49.253062 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:49.253033 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l" Sep 17 15:56:49.253208 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:49.253080 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l" event={"ID":"5b60cad4-7821-4f13-9766-cb303d66bf76","Type":"ContainerDied","Data":"11cc8e06c904a23c36641b40de273b4a4b6ebe719e715c3b4d09d855c1562043"} Sep 17 15:56:49.253208 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:49.253133 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11cc8e06c904a23c36641b40de273b4a4b6ebe719e715c3b4d09d855c1562043" Sep 17 15:56:49.701730 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:49.701695 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:50.529211 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:50.529169 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:55.916425 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:55.916389 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:56.177696 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.177615 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:56.244023 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.243986 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l"] Sep 17 15:56:56.252123 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.252096 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-608b97-xg27l"] Sep 17 15:56:56.266746 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.266721 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:56.538902 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.538867 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" podUID="c6920871-10a8-4c0c-b91a-52234cda4e5c" containerName="main" containerID="cri-o://2266e80b7c6f62f2de597692665b796ba9b1bb98b0e65fb61928a0136ab765e5" gracePeriod=45 Sep 17 15:56:56.631090 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.631059 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:56.735066 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.735034 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:56.792821 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.792767 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:56:56.837392 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.837361 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-home\") pod \"c6920871-10a8-4c0c-b91a-52234cda4e5c\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " Sep 17 15:56:56.837545 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.837420 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6920871-10a8-4c0c-b91a-52234cda4e5c-tls-certs\") pod \"c6920871-10a8-4c0c-b91a-52234cda4e5c\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " Sep 17 15:56:56.837545 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.837472 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdhxh\" (UniqueName: \"kubernetes.io/projected/c6920871-10a8-4c0c-b91a-52234cda4e5c-kube-api-access-fdhxh\") pod \"c6920871-10a8-4c0c-b91a-52234cda4e5c\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " Sep 17 15:56:56.837545 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.837520 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-model-cache\") pod \"c6920871-10a8-4c0c-b91a-52234cda4e5c\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " Sep 17 15:56:56.837713 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.837547 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-tmp-dir\") pod \"c6920871-10a8-4c0c-b91a-52234cda4e5c\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " Sep 17 15:56:56.837713 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.837590 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-dshm\") pod \"c6920871-10a8-4c0c-b91a-52234cda4e5c\" (UID: \"c6920871-10a8-4c0c-b91a-52234cda4e5c\") " Sep 17 15:56:56.837713 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.837600 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-home" (OuterVolumeSpecName: "home") pod "c6920871-10a8-4c0c-b91a-52234cda4e5c" (UID: "c6920871-10a8-4c0c-b91a-52234cda4e5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:56:56.837895 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.837878 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-home\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:56.837971 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.837831 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6920871-10a8-4c0c-b91a-52234cda4e5c" (UID: "c6920871-10a8-4c0c-b91a-52234cda4e5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:56:56.837971 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.837877 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6920871-10a8-4c0c-b91a-52234cda4e5c" (UID: "c6920871-10a8-4c0c-b91a-52234cda4e5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:56:56.839799 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.839777 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-dshm" (OuterVolumeSpecName: "dshm") pod "c6920871-10a8-4c0c-b91a-52234cda4e5c" (UID: "c6920871-10a8-4c0c-b91a-52234cda4e5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:56:56.840003 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.839985 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6920871-10a8-4c0c-b91a-52234cda4e5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6920871-10a8-4c0c-b91a-52234cda4e5c" (UID: "c6920871-10a8-4c0c-b91a-52234cda4e5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:56:56.840003 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.839998 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6920871-10a8-4c0c-b91a-52234cda4e5c-kube-api-access-fdhxh" (OuterVolumeSpecName: "kube-api-access-fdhxh") pod "c6920871-10a8-4c0c-b91a-52234cda4e5c" (UID: "c6920871-10a8-4c0c-b91a-52234cda4e5c"). InnerVolumeSpecName "kube-api-access-fdhxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:56:56.938573 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.938543 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6920871-10a8-4c0c-b91a-52234cda4e5c-tls-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:56.938573 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.938571 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdhxh\" (UniqueName: \"kubernetes.io/projected/c6920871-10a8-4c0c-b91a-52234cda4e5c-kube-api-access-fdhxh\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:56.938938 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.938582 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-model-cache\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:56.938938 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.938593 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-tmp-dir\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:56.938938 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:56.938603 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6920871-10a8-4c0c-b91a-52234cda4e5c-dshm\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:56:57.095579 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.095497 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:57.095579 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.095552 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:57.108395 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.108366 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:57.305140 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.305106 2581 generic.go:356] "Generic (PLEG): container finished" podID="c6920871-10a8-4c0c-b91a-52234cda4e5c" containerID="2266e80b7c6f62f2de597692665b796ba9b1bb98b0e65fb61928a0136ab765e5" exitCode=0 Sep 17 15:56:57.305334 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.305155 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" Sep 17 15:56:57.305334 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.305192 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" event={"ID":"c6920871-10a8-4c0c-b91a-52234cda4e5c","Type":"ContainerDied","Data":"2266e80b7c6f62f2de597692665b796ba9b1bb98b0e65fb61928a0136ab765e5"} Sep 17 15:56:57.305334 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.305235 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws" event={"ID":"c6920871-10a8-4c0c-b91a-52234cda4e5c","Type":"ContainerDied","Data":"794fd4416ee9355ce179407710a05d1a8b651943bbc61fd7809c452ca0ea41a3"} Sep 17 15:56:57.305334 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.305252 2581 scope.go:117] "RemoveContainer" containerID="2266e80b7c6f62f2de597692665b796ba9b1bb98b0e65fb61928a0136ab765e5" Sep 17 15:56:57.314344 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.314327 2581 scope.go:117] "RemoveContainer" containerID="2266e80b7c6f62f2de597692665b796ba9b1bb98b0e65fb61928a0136ab765e5" Sep 17 15:56:57.314665 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:56:57.314632 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2266e80b7c6f62f2de597692665b796ba9b1bb98b0e65fb61928a0136ab765e5\": container with ID starting with 2266e80b7c6f62f2de597692665b796ba9b1bb98b0e65fb61928a0136ab765e5 not found: ID does not exist" containerID="2266e80b7c6f62f2de597692665b796ba9b1bb98b0e65fb61928a0136ab765e5" Sep 17 15:56:57.314765 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.314664 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2266e80b7c6f62f2de597692665b796ba9b1bb98b0e65fb61928a0136ab765e5"} err="failed to get container status \"2266e80b7c6f62f2de597692665b796ba9b1bb98b0e65fb61928a0136ab765e5\": rpc error: code = NotFound desc = could not find container \"2266e80b7c6f62f2de597692665b796ba9b1bb98b0e65fb61928a0136ab765e5\": container with ID starting with 2266e80b7c6f62f2de597692665b796ba9b1bb98b0e65fb61928a0136ab765e5 not found: ID does not exist" Sep 17 15:56:57.319809 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.319788 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:56:57.332065 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.332042 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws"] Sep 17 15:56:57.335322 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.335298 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/auth-test-model-1d63c0-kserve-5cb6478795-ntcws"] Sep 17 15:56:57.462534 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.462462 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b60cad4-7821-4f13-9766-cb303d66bf76" path="/var/lib/kubelet/pods/5b60cad4-7821-4f13-9766-cb303d66bf76/volumes" Sep 17 15:56:57.462844 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.462831 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6920871-10a8-4c0c-b91a-52234cda4e5c" path="/var/lib/kubelet/pods/c6920871-10a8-4c0c-b91a-52234cda4e5c/volumes" Sep 17 15:56:57.657143 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:57.657112 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:56:58.657778 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:56:58.657748 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:57:03.105285 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:03.105254 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:57:03.197702 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:03.197669 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:57:03.283197 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:03.283166 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq"] Sep 17 15:57:03.283402 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:03.283378 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" containerID="cri-o://ee27c59dd121559ebd9a5a122b06ddecba1342681b55705f304f6ee66f2b7fba" gracePeriod=30 Sep 17 15:57:03.305954 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:03.305925 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:57:06.796230 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:06.796146 2581 patch_prober.go:28] interesting pod/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 15:57:06.796230 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:06.796215 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 15:57:08.535292 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.535263 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:57:08.644562 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.644487 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-credential-socket\") pod \"f9ae7f28-a274-491b-8c4d-6c88612e9601\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " Sep 17 15:57:08.644562 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.644530 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-workload-certs\") pod \"f9ae7f28-a274-491b-8c4d-6c88612e9601\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " Sep 17 15:57:08.644562 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.644558 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrjvt\" (UniqueName: \"kubernetes.io/projected/f9ae7f28-a274-491b-8c4d-6c88612e9601-kube-api-access-wrjvt\") pod \"f9ae7f28-a274-491b-8c4d-6c88612e9601\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " Sep 17 15:57:08.644832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.644590 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-token\") pod \"f9ae7f28-a274-491b-8c4d-6c88612e9601\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " Sep 17 15:57:08.644832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.644613 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-data\") pod \"f9ae7f28-a274-491b-8c4d-6c88612e9601\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " Sep 17 15:57:08.644832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.644675 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-workload-socket\") pod \"f9ae7f28-a274-491b-8c4d-6c88612e9601\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " Sep 17 15:57:08.644832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.644701 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-podinfo\") pod \"f9ae7f28-a274-491b-8c4d-6c88612e9601\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " Sep 17 15:57:08.644832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.644719 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ae7f28-a274-491b-8c4d-6c88612e9601-istiod-ca-cert\") pod \"f9ae7f28-a274-491b-8c4d-6c88612e9601\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " Sep 17 15:57:08.644832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.644738 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-envoy\") pod \"f9ae7f28-a274-491b-8c4d-6c88612e9601\" (UID: \"f9ae7f28-a274-491b-8c4d-6c88612e9601\") " Sep 17 15:57:08.644832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.644789 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f9ae7f28-a274-491b-8c4d-6c88612e9601" (UID: "f9ae7f28-a274-491b-8c4d-6c88612e9601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:57:08.644832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.644804 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f9ae7f28-a274-491b-8c4d-6c88612e9601" (UID: "f9ae7f28-a274-491b-8c4d-6c88612e9601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:57:08.645267 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.644984 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-data" (OuterVolumeSpecName: "istio-data") pod "f9ae7f28-a274-491b-8c4d-6c88612e9601" (UID: "f9ae7f28-a274-491b-8c4d-6c88612e9601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:57:08.645267 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.645012 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-credential-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:57:08.645267 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.645056 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-workload-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:57:08.645267 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.645056 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f9ae7f28-a274-491b-8c4d-6c88612e9601" (UID: "f9ae7f28-a274-491b-8c4d-6c88612e9601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:57:08.645267 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.645155 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ae7f28-a274-491b-8c4d-6c88612e9601-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f9ae7f28-a274-491b-8c4d-6c88612e9601" (UID: "f9ae7f28-a274-491b-8c4d-6c88612e9601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 15:57:08.646851 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.646828 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ae7f28-a274-491b-8c4d-6c88612e9601-kube-api-access-wrjvt" (OuterVolumeSpecName: "kube-api-access-wrjvt") pod "f9ae7f28-a274-491b-8c4d-6c88612e9601" (UID: "f9ae7f28-a274-491b-8c4d-6c88612e9601"). InnerVolumeSpecName "kube-api-access-wrjvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:57:08.646978 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.646887 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f9ae7f28-a274-491b-8c4d-6c88612e9601" (UID: "f9ae7f28-a274-491b-8c4d-6c88612e9601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 15:57:08.647182 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.647165 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f9ae7f28-a274-491b-8c4d-6c88612e9601" (UID: "f9ae7f28-a274-491b-8c4d-6c88612e9601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:57:08.647250 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.647182 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-token" (OuterVolumeSpecName: "istio-token") pod "f9ae7f28-a274-491b-8c4d-6c88612e9601" (UID: "f9ae7f28-a274-491b-8c4d-6c88612e9601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:57:08.746274 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.746239 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrjvt\" (UniqueName: \"kubernetes.io/projected/f9ae7f28-a274-491b-8c4d-6c88612e9601-kube-api-access-wrjvt\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:57:08.746274 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.746272 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-token\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:57:08.746274 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.746281 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-data\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:57:08.746456 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.746289 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-workload-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:57:08.746456 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.746297 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-podinfo\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:57:08.746456 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.746305 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ae7f28-a274-491b-8c4d-6c88612e9601-istiod-ca-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:57:08.746456 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:08.746313 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ae7f28-a274-491b-8c4d-6c88612e9601-istio-envoy\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:57:09.371795 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:09.371762 2581 generic.go:356] "Generic (PLEG): container finished" podID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerID="ee27c59dd121559ebd9a5a122b06ddecba1342681b55705f304f6ee66f2b7fba" exitCode=0 Sep 17 15:57:09.371979 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:09.371806 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" Sep 17 15:57:09.371979 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:09.371845 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" event={"ID":"f9ae7f28-a274-491b-8c4d-6c88612e9601","Type":"ContainerDied","Data":"ee27c59dd121559ebd9a5a122b06ddecba1342681b55705f304f6ee66f2b7fba"} Sep 17 15:57:09.371979 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:09.371902 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq" event={"ID":"f9ae7f28-a274-491b-8c4d-6c88612e9601","Type":"ContainerDied","Data":"f624eff8f1f090090d4ea60f9c9d10443962b668dd2d47e0a46a84859f315291"} Sep 17 15:57:09.371979 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:09.371945 2581 scope.go:117] "RemoveContainer" containerID="ee27c59dd121559ebd9a5a122b06ddecba1342681b55705f304f6ee66f2b7fba" Sep 17 15:57:09.382185 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:09.382162 2581 scope.go:117] "RemoveContainer" containerID="7df4b262a8a943d2fcb6bcad2c27dc369033c949a1c7b8a263928bd7be4f0706" Sep 17 15:57:09.390589 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:09.390572 2581 scope.go:117] "RemoveContainer" containerID="ee27c59dd121559ebd9a5a122b06ddecba1342681b55705f304f6ee66f2b7fba" Sep 17 15:57:09.390853 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:57:09.390834 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee27c59dd121559ebd9a5a122b06ddecba1342681b55705f304f6ee66f2b7fba\": container with ID starting with ee27c59dd121559ebd9a5a122b06ddecba1342681b55705f304f6ee66f2b7fba not found: ID does not exist" containerID="ee27c59dd121559ebd9a5a122b06ddecba1342681b55705f304f6ee66f2b7fba" Sep 17 15:57:09.390928 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:09.390862 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee27c59dd121559ebd9a5a122b06ddecba1342681b55705f304f6ee66f2b7fba"} err="failed to get container status \"ee27c59dd121559ebd9a5a122b06ddecba1342681b55705f304f6ee66f2b7fba\": rpc error: code = NotFound desc = could not find container \"ee27c59dd121559ebd9a5a122b06ddecba1342681b55705f304f6ee66f2b7fba\": container with ID starting with ee27c59dd121559ebd9a5a122b06ddecba1342681b55705f304f6ee66f2b7fba not found: ID does not exist" Sep 17 15:57:09.390928 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:09.390880 2581 scope.go:117] "RemoveContainer" containerID="7df4b262a8a943d2fcb6bcad2c27dc369033c949a1c7b8a263928bd7be4f0706" Sep 17 15:57:09.391155 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:57:09.391138 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df4b262a8a943d2fcb6bcad2c27dc369033c949a1c7b8a263928bd7be4f0706\": container with ID starting with 7df4b262a8a943d2fcb6bcad2c27dc369033c949a1c7b8a263928bd7be4f0706 not found: ID does not exist" containerID="7df4b262a8a943d2fcb6bcad2c27dc369033c949a1c7b8a263928bd7be4f0706" Sep 17 15:57:09.391191 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:09.391162 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df4b262a8a943d2fcb6bcad2c27dc369033c949a1c7b8a263928bd7be4f0706"} err="failed to get container status \"7df4b262a8a943d2fcb6bcad2c27dc369033c949a1c7b8a263928bd7be4f0706\": rpc error: code = NotFound desc = could not find container \"7df4b262a8a943d2fcb6bcad2c27dc369033c949a1c7b8a263928bd7be4f0706\": container with ID starting with 7df4b262a8a943d2fcb6bcad2c27dc369033c949a1c7b8a263928bd7be4f0706 not found: ID does not exist" Sep 17 15:57:09.395882 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:09.395861 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq"] Sep 17 15:57:09.405949 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:09.405092 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-608b97-openshift-default-7854df5c76-rhrcq"] Sep 17 15:57:09.461451 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:09.461422 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" path="/var/lib/kubelet/pods/f9ae7f28-a274-491b-8c4d-6c88612e9601/volumes" Sep 17 15:57:36.810087 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:57:36.810053 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:58:10.731103 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.731066 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7791-8def0c"] Sep 17 15:58:10.731561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.731396 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" Sep 17 15:58:10.731561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.731407 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" Sep 17 15:58:10.731561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.731416 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6920871-10a8-4c0c-b91a-52234cda4e5c" containerName="main" Sep 17 15:58:10.731561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.731422 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6920871-10a8-4c0c-b91a-52234cda4e5c" containerName="main" Sep 17 15:58:10.731561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.731430 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b60cad4-7821-4f13-9766-cb303d66bf76" containerName="revoke-keys" Sep 17 15:58:10.731561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.731437 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b60cad4-7821-4f13-9766-cb303d66bf76" containerName="revoke-keys" Sep 17 15:58:10.731561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.731502 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" Sep 17 15:58:10.731561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.731512 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6920871-10a8-4c0c-b91a-52234cda4e5c" containerName="main" Sep 17 15:58:10.731561 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.731519 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b60cad4-7821-4f13-9766-cb303d66bf76" containerName="revoke-keys" Sep 17 15:58:10.734378 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.734361 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7791-8def0c" Sep 17 15:58:10.736702 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.736682 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-55clv\"" Sep 17 15:58:10.741187 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.741162 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7791-8def0c"] Sep 17 15:58:10.872417 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.872386 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbcxw\" (UniqueName: \"kubernetes.io/projected/bb779868-e6e9-4bae-95b7-fa559dd70e77-kube-api-access-gbcxw\") pod \"test-curl-7791-8def0c\" (UID: \"bb779868-e6e9-4bae-95b7-fa559dd70e77\") " pod="opendatahub/test-curl-7791-8def0c" Sep 17 15:58:10.973269 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.973235 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbcxw\" (UniqueName: \"kubernetes.io/projected/bb779868-e6e9-4bae-95b7-fa559dd70e77-kube-api-access-gbcxw\") pod \"test-curl-7791-8def0c\" (UID: \"bb779868-e6e9-4bae-95b7-fa559dd70e77\") " pod="opendatahub/test-curl-7791-8def0c" Sep 17 15:58:10.982603 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:10.982545 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbcxw\" (UniqueName: \"kubernetes.io/projected/bb779868-e6e9-4bae-95b7-fa559dd70e77-kube-api-access-gbcxw\") pod \"test-curl-7791-8def0c\" (UID: \"bb779868-e6e9-4bae-95b7-fa559dd70e77\") " pod="opendatahub/test-curl-7791-8def0c" Sep 17 15:58:11.044728 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:11.044692 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7791-8def0c" Sep 17 15:58:11.353391 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:11.353364 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7791-8def0c"] Sep 17 15:58:11.353936 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:58:11.353897 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb779868_e6e9_4bae_95b7_fa559dd70e77.slice/crio-12ec25595b25dde8a681029e1322d70e73d910171e45ac1e9165c9bdc87e7064 WatchSource:0}: Error finding container 12ec25595b25dde8a681029e1322d70e73d910171e45ac1e9165c9bdc87e7064: Status 404 returned error can't find the container with id 12ec25595b25dde8a681029e1322d70e73d910171e45ac1e9165c9bdc87e7064 Sep 17 15:58:11.691143 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:11.691048 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7791-8def0c" event={"ID":"bb779868-e6e9-4bae-95b7-fa559dd70e77","Type":"ContainerStarted","Data":"7d9d2b2c75f470e5bb1d7ddc1dfdb10a12ff67b65b4d65dbc5f29f0d98f72cb3"} Sep 17 15:58:11.691143 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:11.691099 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7791-8def0c" event={"ID":"bb779868-e6e9-4bae-95b7-fa559dd70e77","Type":"ContainerStarted","Data":"12ec25595b25dde8a681029e1322d70e73d910171e45ac1e9165c9bdc87e7064"} Sep 17 15:58:11.922133 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:11.922082 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7791-8def0c" podStartSLOduration=1.922065816 podStartE2EDuration="1.922065816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:58:11.919403975 +0000 UTC m=+1678.915640721" watchObservedRunningTime="2026-09-17 15:58:11.922065816 +0000 UTC m=+1678.918302562" Sep 17 15:58:12.527645 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.527616 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7791-8def0c"] Sep 17 15:58:12.532222 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.532196 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7791-8def0c"] Sep 17 15:58:12.655717 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.655685 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7791-a687fc"] Sep 17 15:58:12.656044 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.656029 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" Sep 17 15:58:12.656095 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.656045 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" Sep 17 15:58:12.656095 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.656063 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb779868-e6e9-4bae-95b7-fa559dd70e77" containerName="test-curl-7791-8def0c" Sep 17 15:58:12.656095 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.656069 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb779868-e6e9-4bae-95b7-fa559dd70e77" containerName="test-curl-7791-8def0c" Sep 17 15:58:12.656190 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.656120 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb779868-e6e9-4bae-95b7-fa559dd70e77" containerName="test-curl-7791-8def0c" Sep 17 15:58:12.656190 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.656130 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ae7f28-a274-491b-8c4d-6c88612e9601" containerName="istio-proxy" Sep 17 15:58:12.659047 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.659030 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7791-a687fc" Sep 17 15:58:12.667011 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.666989 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7791-a687fc"] Sep 17 15:58:12.696262 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:58:12.696235 2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7791-8def0c" container="test-curl-7791-8def0c" Sep 17 15:58:12.787726 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.787662 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89hwq\" (UniqueName: \"kubernetes.io/projected/fa25da72-9358-400f-88e0-ddcaf35f38f0-kube-api-access-89hwq\") pod \"test-curl-7791-a687fc\" (UID: \"fa25da72-9358-400f-88e0-ddcaf35f38f0\") " pod="opendatahub/test-curl-7791-a687fc" Sep 17 15:58:12.888929 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.888888 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89hwq\" (UniqueName: \"kubernetes.io/projected/fa25da72-9358-400f-88e0-ddcaf35f38f0-kube-api-access-89hwq\") pod \"test-curl-7791-a687fc\" (UID: \"fa25da72-9358-400f-88e0-ddcaf35f38f0\") " pod="opendatahub/test-curl-7791-a687fc" Sep 17 15:58:12.899070 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.899045 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89hwq\" (UniqueName: \"kubernetes.io/projected/fa25da72-9358-400f-88e0-ddcaf35f38f0-kube-api-access-89hwq\") pod \"test-curl-7791-a687fc\" (UID: \"fa25da72-9358-400f-88e0-ddcaf35f38f0\") " pod="opendatahub/test-curl-7791-a687fc" Sep 17 15:58:12.968782 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:12.968755 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7791-a687fc" Sep 17 15:58:13.239843 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:13.239816 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7791-a687fc"] Sep 17 15:58:13.241285 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:58:13.241251 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa25da72_9358_400f_88e0_ddcaf35f38f0.slice/crio-56ebe2b80596545dc3186b41ac50b8474389d11230af4159a3048168521d84f9 WatchSource:0}: Error finding container 56ebe2b80596545dc3186b41ac50b8474389d11230af4159a3048168521d84f9: Status 404 returned error can't find the container with id 56ebe2b80596545dc3186b41ac50b8474389d11230af4159a3048168521d84f9 Sep 17 15:58:13.702747 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:13.702697 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7791-a687fc" event={"ID":"fa25da72-9358-400f-88e0-ddcaf35f38f0","Type":"ContainerStarted","Data":"c450f6420750143aba358e7a8fa7d23c488f00fb8f39a13286b3d55f6f0630d0"} Sep 17 15:58:13.702747 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:13.702755 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7791-a687fc" event={"ID":"fa25da72-9358-400f-88e0-ddcaf35f38f0","Type":"ContainerStarted","Data":"56ebe2b80596545dc3186b41ac50b8474389d11230af4159a3048168521d84f9"} Sep 17 15:58:13.703037 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:13.702843 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7791-8def0c" podUID="bb779868-e6e9-4bae-95b7-fa559dd70e77" containerName="test-curl-7791-8def0c" containerID="cri-o://7d9d2b2c75f470e5bb1d7ddc1dfdb10a12ff67b65b4d65dbc5f29f0d98f72cb3" gracePeriod=2 Sep 17 15:58:13.719477 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:13.719430 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7791-a687fc" podStartSLOduration=1.719415023 podStartE2EDuration="1.719415023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:58:13.717398462 +0000 UTC m=+1680.713635208" watchObservedRunningTime="2026-09-17 15:58:13.719415023 +0000 UTC m=+1680.715651818" Sep 17 15:58:14.322113 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:14.322080 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7791-a687fc"] Sep 17 15:58:14.333676 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:14.331289 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7791-a687fc"] Sep 17 15:58:14.708236 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:58:14.708204 2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7791-a687fc" container="test-curl-7791-a687fc" Sep 17 15:58:15.714033 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:15.714000 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7791-a687fc" podUID="fa25da72-9358-400f-88e0-ddcaf35f38f0" containerName="test-curl-7791-a687fc" containerID="cri-o://c450f6420750143aba358e7a8fa7d23c488f00fb8f39a13286b3d55f6f0630d0" gracePeriod=2 Sep 17 15:58:15.861130 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:15.861106 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7791-8def0c" Sep 17 15:58:16.018830 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:16.018750 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbcxw\" (UniqueName: \"kubernetes.io/projected/bb779868-e6e9-4bae-95b7-fa559dd70e77-kube-api-access-gbcxw\") pod \"bb779868-e6e9-4bae-95b7-fa559dd70e77\" (UID: \"bb779868-e6e9-4bae-95b7-fa559dd70e77\") " Sep 17 15:58:16.020750 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:16.020720 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb779868-e6e9-4bae-95b7-fa559dd70e77-kube-api-access-gbcxw" (OuterVolumeSpecName: "kube-api-access-gbcxw") pod "bb779868-e6e9-4bae-95b7-fa559dd70e77" (UID: "bb779868-e6e9-4bae-95b7-fa559dd70e77"). InnerVolumeSpecName "kube-api-access-gbcxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:58:16.119749 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:16.119712 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbcxw\" (UniqueName: \"kubernetes.io/projected/bb779868-e6e9-4bae-95b7-fa559dd70e77-kube-api-access-gbcxw\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:58:16.355020 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:16.354985 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq"] Sep 17 15:58:16.376013 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:16.375971 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:58:16.720006 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:16.719898 2581 generic.go:356] "Generic (PLEG): container finished" podID="bb779868-e6e9-4bae-95b7-fa559dd70e77" containerID="7d9d2b2c75f470e5bb1d7ddc1dfdb10a12ff67b65b4d65dbc5f29f0d98f72cb3" exitCode=137 Sep 17 15:58:16.720006 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:16.719956 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7791-8def0c" Sep 17 15:58:16.720454 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:16.720007 2581 scope.go:117] "RemoveContainer" containerID="7d9d2b2c75f470e5bb1d7ddc1dfdb10a12ff67b65b4d65dbc5f29f0d98f72cb3" Sep 17 15:58:16.729513 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:16.729492 2581 scope.go:117] "RemoveContainer" containerID="7d9d2b2c75f470e5bb1d7ddc1dfdb10a12ff67b65b4d65dbc5f29f0d98f72cb3" Sep 17 15:58:16.729805 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:58:16.729781 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9d2b2c75f470e5bb1d7ddc1dfdb10a12ff67b65b4d65dbc5f29f0d98f72cb3\": container with ID starting with 7d9d2b2c75f470e5bb1d7ddc1dfdb10a12ff67b65b4d65dbc5f29f0d98f72cb3 not found: ID does not exist" containerID="7d9d2b2c75f470e5bb1d7ddc1dfdb10a12ff67b65b4d65dbc5f29f0d98f72cb3" Sep 17 15:58:16.729870 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:16.729820 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9d2b2c75f470e5bb1d7ddc1dfdb10a12ff67b65b4d65dbc5f29f0d98f72cb3"} err="failed to get container status \"7d9d2b2c75f470e5bb1d7ddc1dfdb10a12ff67b65b4d65dbc5f29f0d98f72cb3\": rpc error: code = NotFound desc = could not find container \"7d9d2b2c75f470e5bb1d7ddc1dfdb10a12ff67b65b4d65dbc5f29f0d98f72cb3\": container with ID starting with 7d9d2b2c75f470e5bb1d7ddc1dfdb10a12ff67b65b4d65dbc5f29f0d98f72cb3 not found: ID does not exist" Sep 17 15:58:16.730220 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:58:16.730196 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7791-8def0c\" is forbidden: User \"system:node:ip-10-0-128-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-15.ec2.internal' and this object" podUID="bb779868-e6e9-4bae-95b7-fa559dd70e77" pod="opendatahub/test-curl-7791-8def0c" Sep 17 15:58:17.293232 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:17.293203 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:58:17.462391 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:17.462361 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb779868-e6e9-4bae-95b7-fa559dd70e77" path="/var/lib/kubelet/pods/bb779868-e6e9-4bae-95b7-fa559dd70e77/volumes" Sep 17 15:58:17.887822 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:17.887801 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7791-a687fc" Sep 17 15:58:18.035508 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:18.035486 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89hwq\" (UniqueName: \"kubernetes.io/projected/fa25da72-9358-400f-88e0-ddcaf35f38f0-kube-api-access-89hwq\") pod \"fa25da72-9358-400f-88e0-ddcaf35f38f0\" (UID: \"fa25da72-9358-400f-88e0-ddcaf35f38f0\") " Sep 17 15:58:18.037479 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:18.037457 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa25da72-9358-400f-88e0-ddcaf35f38f0-kube-api-access-89hwq" (OuterVolumeSpecName: "kube-api-access-89hwq") pod "fa25da72-9358-400f-88e0-ddcaf35f38f0" (UID: "fa25da72-9358-400f-88e0-ddcaf35f38f0"). InnerVolumeSpecName "kube-api-access-89hwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:58:18.136532 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:18.136508 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89hwq\" (UniqueName: \"kubernetes.io/projected/fa25da72-9358-400f-88e0-ddcaf35f38f0-kube-api-access-89hwq\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:58:18.732466 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:18.732428 2581 generic.go:356] "Generic (PLEG): container finished" podID="fa25da72-9358-400f-88e0-ddcaf35f38f0" containerID="c450f6420750143aba358e7a8fa7d23c488f00fb8f39a13286b3d55f6f0630d0" exitCode=137 Sep 17 15:58:18.732649 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:18.732474 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7791-a687fc" Sep 17 15:58:18.732649 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:18.732479 2581 scope.go:117] "RemoveContainer" containerID="c450f6420750143aba358e7a8fa7d23c488f00fb8f39a13286b3d55f6f0630d0" Sep 17 15:58:18.741822 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:18.741803 2581 scope.go:117] "RemoveContainer" containerID="c450f6420750143aba358e7a8fa7d23c488f00fb8f39a13286b3d55f6f0630d0" Sep 17 15:58:18.742099 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:58:18.742078 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c450f6420750143aba358e7a8fa7d23c488f00fb8f39a13286b3d55f6f0630d0\": container with ID starting with c450f6420750143aba358e7a8fa7d23c488f00fb8f39a13286b3d55f6f0630d0 not found: ID does not exist" containerID="c450f6420750143aba358e7a8fa7d23c488f00fb8f39a13286b3d55f6f0630d0" Sep 17 15:58:18.742194 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:18.742106 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c450f6420750143aba358e7a8fa7d23c488f00fb8f39a13286b3d55f6f0630d0"} err="failed to get container status \"c450f6420750143aba358e7a8fa7d23c488f00fb8f39a13286b3d55f6f0630d0\": rpc error: code = NotFound desc = could not find container \"c450f6420750143aba358e7a8fa7d23c488f00fb8f39a13286b3d55f6f0630d0\": container with ID starting with c450f6420750143aba358e7a8fa7d23c488f00fb8f39a13286b3d55f6f0630d0 not found: ID does not exist" Sep 17 15:58:18.742818 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:58:18.742796 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7791-a687fc\" is forbidden: User \"system:node:ip-10-0-128-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-15.ec2.internal' and this object" podUID="fa25da72-9358-400f-88e0-ddcaf35f38f0" pod="opendatahub/test-curl-7791-a687fc" Sep 17 15:58:19.462280 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:19.462246 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa25da72-9358-400f-88e0-ddcaf35f38f0" path="/var/lib/kubelet/pods/fa25da72-9358-400f-88e0-ddcaf35f38f0/volumes" Sep 17 15:58:20.788519 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:20.788488 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:58:21.469008 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:21.468973 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:58:31.373428 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.373368 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" podUID="96a16355-2030-4db0-bab7-69805bf1ae66" containerName="main" containerID="cri-o://14471cbf566687ad44e84e1c04cdbe22217fa33cdf65f67ad7a957c0e2f3fe2a" gracePeriod=45 Sep 17 15:58:31.620602 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.620572 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:58:31.639738 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.639671 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-tmp-dir\") pod \"96a16355-2030-4db0-bab7-69805bf1ae66\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " Sep 17 15:58:31.639738 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.639723 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-dshm\") pod \"96a16355-2030-4db0-bab7-69805bf1ae66\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " Sep 17 15:58:31.639957 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.639760 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a16355-2030-4db0-bab7-69805bf1ae66-tls-certs\") pod \"96a16355-2030-4db0-bab7-69805bf1ae66\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " Sep 17 15:58:31.639957 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.639794 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-home\") pod \"96a16355-2030-4db0-bab7-69805bf1ae66\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " Sep 17 15:58:31.639957 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.639820 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-model-cache\") pod \"96a16355-2030-4db0-bab7-69805bf1ae66\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " Sep 17 15:58:31.639957 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.639889 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x5z6\" (UniqueName: \"kubernetes.io/projected/96a16355-2030-4db0-bab7-69805bf1ae66-kube-api-access-6x5z6\") pod \"96a16355-2030-4db0-bab7-69805bf1ae66\" (UID: \"96a16355-2030-4db0-bab7-69805bf1ae66\") " Sep 17 15:58:31.640146 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.639954 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96a16355-2030-4db0-bab7-69805bf1ae66" (UID: "96a16355-2030-4db0-bab7-69805bf1ae66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:58:31.640146 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.640044 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-home" (OuterVolumeSpecName: "home") pod "96a16355-2030-4db0-bab7-69805bf1ae66" (UID: "96a16355-2030-4db0-bab7-69805bf1ae66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:58:31.640146 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.640123 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-model-cache" (OuterVolumeSpecName: "model-cache") pod "96a16355-2030-4db0-bab7-69805bf1ae66" (UID: "96a16355-2030-4db0-bab7-69805bf1ae66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:58:31.640265 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.640238 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-home\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:58:31.640315 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.640274 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-model-cache\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:58:31.640315 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.640289 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-tmp-dir\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:58:31.641729 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.641709 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-dshm" (OuterVolumeSpecName: "dshm") pod "96a16355-2030-4db0-bab7-69805bf1ae66" (UID: "96a16355-2030-4db0-bab7-69805bf1ae66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:58:31.641862 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.641843 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a16355-2030-4db0-bab7-69805bf1ae66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96a16355-2030-4db0-bab7-69805bf1ae66" (UID: "96a16355-2030-4db0-bab7-69805bf1ae66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:58:31.641939 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.641878 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a16355-2030-4db0-bab7-69805bf1ae66-kube-api-access-6x5z6" (OuterVolumeSpecName: "kube-api-access-6x5z6") pod "96a16355-2030-4db0-bab7-69805bf1ae66" (UID: "96a16355-2030-4db0-bab7-69805bf1ae66"). InnerVolumeSpecName "kube-api-access-6x5z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:58:31.741005 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.740972 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x5z6\" (UniqueName: \"kubernetes.io/projected/96a16355-2030-4db0-bab7-69805bf1ae66-kube-api-access-6x5z6\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:58:31.741005 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.741001 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a16355-2030-4db0-bab7-69805bf1ae66-dshm\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:58:31.741005 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.741011 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a16355-2030-4db0-bab7-69805bf1ae66-tls-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:58:31.812993 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.812958 2581 generic.go:356] "Generic (PLEG): container finished" podID="96a16355-2030-4db0-bab7-69805bf1ae66" containerID="14471cbf566687ad44e84e1c04cdbe22217fa33cdf65f67ad7a957c0e2f3fe2a" exitCode=0 Sep 17 15:58:31.813133 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.813006 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" Sep 17 15:58:31.813133 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.813044 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" event={"ID":"96a16355-2030-4db0-bab7-69805bf1ae66","Type":"ContainerDied","Data":"14471cbf566687ad44e84e1c04cdbe22217fa33cdf65f67ad7a957c0e2f3fe2a"} Sep 17 15:58:31.813133 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.813084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq" event={"ID":"96a16355-2030-4db0-bab7-69805bf1ae66","Type":"ContainerDied","Data":"eb8cf132ad67a3b0bdfdada46555da0144454568112bb21f64fa13f56de8f1a8"} Sep 17 15:58:31.813133 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.813106 2581 scope.go:117] "RemoveContainer" containerID="14471cbf566687ad44e84e1c04cdbe22217fa33cdf65f67ad7a957c0e2f3fe2a" Sep 17 15:58:31.822458 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.822439 2581 scope.go:117] "RemoveContainer" containerID="14471cbf566687ad44e84e1c04cdbe22217fa33cdf65f67ad7a957c0e2f3fe2a" Sep 17 15:58:31.822743 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:58:31.822724 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14471cbf566687ad44e84e1c04cdbe22217fa33cdf65f67ad7a957c0e2f3fe2a\": container with ID starting with 14471cbf566687ad44e84e1c04cdbe22217fa33cdf65f67ad7a957c0e2f3fe2a not found: ID does not exist" containerID="14471cbf566687ad44e84e1c04cdbe22217fa33cdf65f67ad7a957c0e2f3fe2a" Sep 17 15:58:31.822796 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.822752 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14471cbf566687ad44e84e1c04cdbe22217fa33cdf65f67ad7a957c0e2f3fe2a"} err="failed to get container status \"14471cbf566687ad44e84e1c04cdbe22217fa33cdf65f67ad7a957c0e2f3fe2a\": rpc error: code = NotFound desc = could not find container \"14471cbf566687ad44e84e1c04cdbe22217fa33cdf65f67ad7a957c0e2f3fe2a\": container with ID starting with 14471cbf566687ad44e84e1c04cdbe22217fa33cdf65f67ad7a957c0e2f3fe2a not found: ID does not exist" Sep 17 15:58:31.842232 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.842206 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq"] Sep 17 15:58:31.849646 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:31.849625 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d63c0/sub-test-model-1d63c0-kserve-688cdb4d69-z9fnq"] Sep 17 15:58:33.462510 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:33.462477 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a16355-2030-4db0-bab7-69805bf1ae66" path="/var/lib/kubelet/pods/96a16355-2030-4db0-bab7-69805bf1ae66/volumes" Sep 17 15:58:41.488428 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:41.488346 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:58:41.567194 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:41.567155 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:58:50.363708 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.363675 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9"] Sep 17 15:58:50.364100 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.364085 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a16355-2030-4db0-bab7-69805bf1ae66" containerName="main" Sep 17 15:58:50.364100 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.364101 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a16355-2030-4db0-bab7-69805bf1ae66" containerName="main" Sep 17 15:58:50.364177 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.364116 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa25da72-9358-400f-88e0-ddcaf35f38f0" containerName="test-curl-7791-a687fc" Sep 17 15:58:50.364177 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.364122 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa25da72-9358-400f-88e0-ddcaf35f38f0" containerName="test-curl-7791-a687fc" Sep 17 15:58:50.364177 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.364172 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a16355-2030-4db0-bab7-69805bf1ae66" containerName="main" Sep 17 15:58:50.364274 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.364180 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa25da72-9358-400f-88e0-ddcaf35f38f0" containerName="test-curl-7791-a687fc" Sep 17 15:58:50.366964 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.366948 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.370103 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.370060 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5818c9\"/\"default-dockercfg-dxwws\"" Sep 17 15:58:50.370103 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.370082 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5818c9\"/\"rate-test-model-5818c9-kserve-self-signed-certs\"" Sep 17 15:58:50.370272 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.370129 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5818c9\"/\"kube-root-ca.crt\"" Sep 17 15:58:50.370272 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.370083 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5818c9\"/\"openshift-service-ca.crt\"" Sep 17 15:58:50.380130 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.380109 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9"] Sep 17 15:58:50.396317 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.396296 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:58:50.496501 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.496465 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78kch\" (UniqueName: \"kubernetes.io/projected/65465fcf-2950-4260-8efa-7592dba5dd7d-kube-api-access-78kch\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.496686 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.496510 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-dshm\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.496686 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.496601 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-model-cache\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.496686 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.496653 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-home\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.496686 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.496676 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65465fcf-2950-4260-8efa-7592dba5dd7d-tls-certs\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.496861 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.496702 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-tmp-dir\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.597953 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.597899 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-model-cache\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.598120 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.597962 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-home\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.598120 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.598063 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65465fcf-2950-4260-8efa-7592dba5dd7d-tls-certs\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.598120 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.598101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-tmp-dir\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.598289 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.598129 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78kch\" (UniqueName: \"kubernetes.io/projected/65465fcf-2950-4260-8efa-7592dba5dd7d-kube-api-access-78kch\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.598289 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.598160 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-dshm\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.598404 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.598336 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-home\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.598459 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.598394 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-model-cache\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.598459 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.598407 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-tmp-dir\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.600334 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.600311 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-dshm\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.600598 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.600582 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65465fcf-2950-4260-8efa-7592dba5dd7d-tls-certs\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.607470 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.607447 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78kch\" (UniqueName: \"kubernetes.io/projected/65465fcf-2950-4260-8efa-7592dba5dd7d-kube-api-access-78kch\") pod \"rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.676068 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.676001 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:58:50.951544 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:50.951520 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9"] Sep 17 15:58:50.952944 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:58:50.952897 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65465fcf_2950_4260_8efa_7592dba5dd7d.slice/crio-c33d691b254ba799d2c6d44617b6f250a0244f777e1816e14e08bc6301eb1417 WatchSource:0}: Error finding container c33d691b254ba799d2c6d44617b6f250a0244f777e1816e14e08bc6301eb1417: Status 404 returned error can't find the container with id c33d691b254ba799d2c6d44617b6f250a0244f777e1816e14e08bc6301eb1417 Sep 17 15:58:51.918622 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:51.918584 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" event={"ID":"65465fcf-2950-4260-8efa-7592dba5dd7d","Type":"ContainerStarted","Data":"987c8b6c3ae48ee5868baba6b1059f016a525c9c7f4363273d55b79e7ef3cb8b"} Sep 17 15:58:51.919015 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:51.918632 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" event={"ID":"65465fcf-2950-4260-8efa-7592dba5dd7d","Type":"ContainerStarted","Data":"c33d691b254ba799d2c6d44617b6f250a0244f777e1816e14e08bc6301eb1417"} Sep 17 15:58:51.941504 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:58:51.941456 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" podStartSLOduration=1.941436465 podStartE2EDuration="1.941436465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:58:51.938977781 +0000 UTC m=+1718.935214528" watchObservedRunningTime="2026-09-17 15:58:51.941436465 +0000 UTC m=+1718.937673251" Sep 17 15:59:00.676856 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:00.676824 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:59:00.677266 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:00.676925 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:59:00.689231 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:00.689207 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:59:00.975462 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:00.975386 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:59:03.733491 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:03.733455 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:31.576640 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:31.576606 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:32.483597 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:32.483565 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9"] Sep 17 15:59:32.496587 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:32.496561 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:32.978554 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:32.978520 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:33.321704 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.321671 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb"] Sep 17 15:59:33.325159 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.325143 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.332233 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.332208 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-420219-openshift-default-dockercfg-fdmls\"" Sep 17 15:59:33.347029 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.347005 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb"] Sep 17 15:59:33.352163 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.352141 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-credential-socket\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.352303 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.352184 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istiod-ca-cert\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.352303 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.352266 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-workload-socket\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.352303 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.352293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-workload-certs\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.352482 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.352329 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-podinfo\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.352482 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.352356 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-data\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.352482 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.352401 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-envoy\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.352482 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.352454 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-token\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.352650 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.352508 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9s45\" (UniqueName: \"kubernetes.io/projected/cfb89bed-f577-4a9d-bde2-0e388ab3669a-kube-api-access-n9s45\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.412805 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.412771 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:33.453831 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.453802 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-workload-socket\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.453831 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.453829 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-workload-certs\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.454076 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.453853 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-podinfo\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.454076 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.453870 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-data\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.454076 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.453959 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-envoy\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.454076 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.454000 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-token\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.454076 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.454052 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9s45\" (UniqueName: \"kubernetes.io/projected/cfb89bed-f577-4a9d-bde2-0e388ab3669a-kube-api-access-n9s45\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.454339 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.454078 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-credential-socket\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.454339 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.454119 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istiod-ca-cert\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.454339 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.454277 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-workload-certs\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.454339 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.454311 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-data\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.454536 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.454365 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-workload-socket\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.454596 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.454575 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-credential-socket\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.454749 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.454720 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istiod-ca-cert\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.456351 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.456326 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-envoy\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.456524 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.456506 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-podinfo\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.478248 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.478227 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9s45\" (UniqueName: \"kubernetes.io/projected/cfb89bed-f577-4a9d-bde2-0e388ab3669a-kube-api-access-n9s45\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.489777 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.489754 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-token\") pod \"e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.635691 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.635603 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:33.656296 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.656268 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:33.927690 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.927605 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb"] Sep 17 15:59:33.934289 ip-10-0-128-15 kubenswrapper[2581]: W0917 15:59:33.934260 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb89bed_f577_4a9d_bde2_0e388ab3669a.slice/crio-0fac493c944fe075edd17ea394c56dc9834058b2d620bc4148ce0d0434bc7291 WatchSource:0}: Error finding container 0fac493c944fe075edd17ea394c56dc9834058b2d620bc4148ce0d0434bc7291: Status 404 returned error can't find the container with id 0fac493c944fe075edd17ea394c56dc9834058b2d620bc4148ce0d0434bc7291 Sep 17 15:59:33.936302 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.936273 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:59:33.936393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.936353 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:59:33.936435 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:33.936391 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:59:34.140134 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:34.140084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" event={"ID":"cfb89bed-f577-4a9d-bde2-0e388ab3669a","Type":"ContainerStarted","Data":"eb4a2079d5f82fb2deddac30723ed861fc1714aa6f2dbe83d36147b161d48bf1"} Sep 17 15:59:34.140518 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:34.140161 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" event={"ID":"cfb89bed-f577-4a9d-bde2-0e388ab3669a","Type":"ContainerStarted","Data":"0fac493c944fe075edd17ea394c56dc9834058b2d620bc4148ce0d0434bc7291"} Sep 17 15:59:34.635735 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:34.635704 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:35.530373 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:35.530339 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:35.641465 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:35.641438 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:35.673727 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:35.673676 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" podStartSLOduration=2.673661051 podStartE2EDuration="2.673661051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:34.187019659 +0000 UTC m=+1761.183256404" watchObservedRunningTime="2026-09-17 15:59:35.673661051 +0000 UTC m=+1762.669897797" Sep 17 15:59:36.150592 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:36.150561 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:36.151469 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:36.151449 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:36.247425 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:36.247391 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:36.349615 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:36.349585 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:36.616713 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:36.616684 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:37.820466 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:37.820434 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:38.756419 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.756379 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf"] Sep 17 15:59:38.760067 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.760050 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:38.763414 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.763393 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-420219-serving-cert\"" Sep 17 15:59:38.763560 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.763505 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mdtj9\"" Sep 17 15:59:38.764176 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.764154 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 15:59:38.764289 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.764235 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 15:59:38.765865 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.765847 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 15:59:38.777832 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.777807 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf"] Sep 17 15:59:38.801846 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.801815 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rtl8\" (UniqueName: \"kubernetes.io/projected/cfb77b55-a1b3-4416-96cf-a0a7854404e7-kube-api-access-5rtl8\") pod \"maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:38.801985 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.801864 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb77b55-a1b3-4416-96cf-a0a7854404e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:38.802038 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.801995 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb77b55-a1b3-4416-96cf-a0a7854404e7-metrics-certs\") pod \"maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:38.802087 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.802047 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb77b55-a1b3-4416-96cf-a0a7854404e7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:38.807304 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.807279 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:38.903170 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.903135 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb77b55-a1b3-4416-96cf-a0a7854404e7-metrics-certs\") pod \"maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:38.903545 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.903188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb77b55-a1b3-4416-96cf-a0a7854404e7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:38.903545 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.903218 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rtl8\" (UniqueName: \"kubernetes.io/projected/cfb77b55-a1b3-4416-96cf-a0a7854404e7-kube-api-access-5rtl8\") pod \"maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:38.903545 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.903243 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb77b55-a1b3-4416-96cf-a0a7854404e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:38.903885 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.903865 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb77b55-a1b3-4416-96cf-a0a7854404e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:38.905765 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.905743 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb77b55-a1b3-4416-96cf-a0a7854404e7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:38.905879 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.905793 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb77b55-a1b3-4416-96cf-a0a7854404e7-metrics-certs\") pod \"maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:38.913175 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.913147 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rtl8\" (UniqueName: \"kubernetes.io/projected/cfb77b55-a1b3-4416-96cf-a0a7854404e7-kube-api-access-5rtl8\") pod \"maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:38.920350 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:38.920236 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:39.071776 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:39.071747 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:39.349347 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:39.349324 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf"] Sep 17 15:59:39.355100 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:39.355075 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 15:59:39.369850 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:39.369829 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:39.474201 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:39.474167 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 15:59:40.172161 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:40.172125 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" event={"ID":"cfb77b55-a1b3-4416-96cf-a0a7854404e7","Type":"ContainerStarted","Data":"9be29c41e3401719c5a2cd7bd02d497291ad548815b15fee7bad23bc0858088a"} Sep 17 15:59:40.172578 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:40.172173 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" event={"ID":"cfb77b55-a1b3-4416-96cf-a0a7854404e7","Type":"ContainerStarted","Data":"709943536b749711caeb6b2184b30e9f4cfe6825990661c775d05515d0d54ac8"} Sep 17 15:59:40.172578 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:40.172260 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:40.199543 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:40.199498 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" podStartSLOduration=1.583198208 podStartE2EDuration="2.199481676s" totalImagesPullingTime="616.283468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:40.197963806 +0000 UTC m=+1767.194200553" watchObservedRunningTime="2026-09-17 15:59:40.199481676 +0000 UTC m=+1767.195718422" Sep 17 15:59:46.188318 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:46.188278 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 15:59:47.502441 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.502394 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" podUID="65465fcf-2950-4260-8efa-7592dba5dd7d" containerName="main" containerID="cri-o://987c8b6c3ae48ee5868baba6b1059f016a525c9c7f4363273d55b79e7ef3cb8b" gracePeriod=45 Sep 17 15:59:47.767772 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.767748 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:59:47.898939 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.898087 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-model-cache\") pod \"65465fcf-2950-4260-8efa-7592dba5dd7d\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " Sep 17 15:59:47.898939 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.898221 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-home\") pod \"65465fcf-2950-4260-8efa-7592dba5dd7d\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " Sep 17 15:59:47.898939 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.898252 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-tmp-dir\") pod \"65465fcf-2950-4260-8efa-7592dba5dd7d\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " Sep 17 15:59:47.898939 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.898276 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65465fcf-2950-4260-8efa-7592dba5dd7d-tls-certs\") pod \"65465fcf-2950-4260-8efa-7592dba5dd7d\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " Sep 17 15:59:47.898939 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.898306 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-dshm\") pod \"65465fcf-2950-4260-8efa-7592dba5dd7d\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " Sep 17 15:59:47.898939 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.898354 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78kch\" (UniqueName: \"kubernetes.io/projected/65465fcf-2950-4260-8efa-7592dba5dd7d-kube-api-access-78kch\") pod \"65465fcf-2950-4260-8efa-7592dba5dd7d\" (UID: \"65465fcf-2950-4260-8efa-7592dba5dd7d\") " Sep 17 15:59:47.898939 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.898830 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "65465fcf-2950-4260-8efa-7592dba5dd7d" (UID: "65465fcf-2950-4260-8efa-7592dba5dd7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:59:47.899477 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.899112 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-home" (OuterVolumeSpecName: "home") pod "65465fcf-2950-4260-8efa-7592dba5dd7d" (UID: "65465fcf-2950-4260-8efa-7592dba5dd7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:59:47.899477 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.899300 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-model-cache\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:59:47.899477 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.899329 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-home\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:59:47.899477 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.899371 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65465fcf-2950-4260-8efa-7592dba5dd7d" (UID: "65465fcf-2950-4260-8efa-7592dba5dd7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:59:47.901164 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.901134 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-dshm" (OuterVolumeSpecName: "dshm") pod "65465fcf-2950-4260-8efa-7592dba5dd7d" (UID: "65465fcf-2950-4260-8efa-7592dba5dd7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 15:59:47.901164 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.901150 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65465fcf-2950-4260-8efa-7592dba5dd7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65465fcf-2950-4260-8efa-7592dba5dd7d" (UID: "65465fcf-2950-4260-8efa-7592dba5dd7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 15:59:47.901309 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.901225 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65465fcf-2950-4260-8efa-7592dba5dd7d-kube-api-access-78kch" (OuterVolumeSpecName: "kube-api-access-78kch") pod "65465fcf-2950-4260-8efa-7592dba5dd7d" (UID: "65465fcf-2950-4260-8efa-7592dba5dd7d"). InnerVolumeSpecName "kube-api-access-78kch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 15:59:47.999790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.999755 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-tmp-dir\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:59:47.999790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.999784 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65465fcf-2950-4260-8efa-7592dba5dd7d-tls-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:59:47.999790 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.999795 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65465fcf-2950-4260-8efa-7592dba5dd7d-dshm\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:59:48.000060 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:47.999805 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78kch\" (UniqueName: \"kubernetes.io/projected/65465fcf-2950-4260-8efa-7592dba5dd7d-kube-api-access-78kch\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 15:59:48.219338 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:48.219299 2581 generic.go:356] "Generic (PLEG): container finished" podID="65465fcf-2950-4260-8efa-7592dba5dd7d" containerID="987c8b6c3ae48ee5868baba6b1059f016a525c9c7f4363273d55b79e7ef3cb8b" exitCode=0 Sep 17 15:59:48.219338 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:48.219338 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" Sep 17 15:59:48.219558 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:48.219388 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" event={"ID":"65465fcf-2950-4260-8efa-7592dba5dd7d","Type":"ContainerDied","Data":"987c8b6c3ae48ee5868baba6b1059f016a525c9c7f4363273d55b79e7ef3cb8b"} Sep 17 15:59:48.219558 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:48.219428 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9" event={"ID":"65465fcf-2950-4260-8efa-7592dba5dd7d","Type":"ContainerDied","Data":"c33d691b254ba799d2c6d44617b6f250a0244f777e1816e14e08bc6301eb1417"} Sep 17 15:59:48.219558 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:48.219446 2581 scope.go:117] "RemoveContainer" containerID="987c8b6c3ae48ee5868baba6b1059f016a525c9c7f4363273d55b79e7ef3cb8b" Sep 17 15:59:48.228996 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:48.228978 2581 scope.go:117] "RemoveContainer" containerID="987c8b6c3ae48ee5868baba6b1059f016a525c9c7f4363273d55b79e7ef3cb8b" Sep 17 15:59:48.229262 ip-10-0-128-15 kubenswrapper[2581]: E0917 15:59:48.229243 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987c8b6c3ae48ee5868baba6b1059f016a525c9c7f4363273d55b79e7ef3cb8b\": container with ID starting with 987c8b6c3ae48ee5868baba6b1059f016a525c9c7f4363273d55b79e7ef3cb8b not found: ID does not exist" containerID="987c8b6c3ae48ee5868baba6b1059f016a525c9c7f4363273d55b79e7ef3cb8b" Sep 17 15:59:48.229327 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:48.229270 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987c8b6c3ae48ee5868baba6b1059f016a525c9c7f4363273d55b79e7ef3cb8b"} err="failed to get container status \"987c8b6c3ae48ee5868baba6b1059f016a525c9c7f4363273d55b79e7ef3cb8b\": rpc error: code = NotFound desc = could not find container \"987c8b6c3ae48ee5868baba6b1059f016a525c9c7f4363273d55b79e7ef3cb8b\": container with ID starting with 987c8b6c3ae48ee5868baba6b1059f016a525c9c7f4363273d55b79e7ef3cb8b not found: ID does not exist" Sep 17 15:59:48.261451 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:48.261424 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9"] Sep 17 15:59:48.277440 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:48.277402 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/rate-test-model-5818c9-kserve-7cd9b9b7ff-9kvf9"] Sep 17 15:59:49.461952 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:49.461906 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65465fcf-2950-4260-8efa-7592dba5dd7d" path="/var/lib/kubelet/pods/65465fcf-2950-4260-8efa-7592dba5dd7d/volumes" Sep 17 15:59:52.192037 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:52.191999 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 17 15:59:52.192501 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:52.192069 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 15:59:53.249997 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:53.249965 2581 generic.go:356] "Generic (PLEG): container finished" podID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerID="eb4a2079d5f82fb2deddac30723ed861fc1714aa6f2dbe83d36147b161d48bf1" exitCode=0 Sep 17 15:59:53.250401 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:53.250044 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" event={"ID":"cfb89bed-f577-4a9d-bde2-0e388ab3669a","Type":"ContainerDied","Data":"eb4a2079d5f82fb2deddac30723ed861fc1714aa6f2dbe83d36147b161d48bf1"} Sep 17 15:59:53.250591 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:53.250424 2581 scope.go:117] "RemoveContainer" containerID="eb4a2079d5f82fb2deddac30723ed861fc1714aa6f2dbe83d36147b161d48bf1" Sep 17 15:59:53.251405 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:53.251378 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:59:53.251456 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:53.251448 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:59:53.251491 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:53.251482 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 15:59:54.257307 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:54.257270 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" event={"ID":"cfb89bed-f577-4a9d-bde2-0e388ab3669a","Type":"ContainerStarted","Data":"4560bc6bddc9ac35c7e91c82d04b727dfdb3b7576ba1a6028a625464075b4ebc"} Sep 17 15:59:54.635990 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:54.635947 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:54.636210 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:54.636185 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Sep 17 15:59:54.636268 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:54.636249 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Sep 17 15:59:55.636143 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:55.636104 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Sep 17 15:59:55.636560 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:55.636195 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Sep 17 15:59:56.636877 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:56.636843 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Sep 17 15:59:56.637297 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:56.636898 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Sep 17 15:59:57.636393 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:57.636347 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Sep 17 15:59:57.636582 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:57.636413 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Sep 17 15:59:58.636484 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:58.636448 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Sep 17 15:59:58.636865 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:58.636507 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Sep 17 15:59:59.639821 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:59.639790 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:59.640295 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:59.640068 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 15:59:59.640819 ip-10-0-128-15 kubenswrapper[2581]: I0917 15:59:59.640803 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 16:00:00.147069 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:00.147035 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9"] Sep 17 16:00:00.147398 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:00.147386 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65465fcf-2950-4260-8efa-7592dba5dd7d" containerName="main" Sep 17 16:00:00.147452 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:00.147399 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="65465fcf-2950-4260-8efa-7592dba5dd7d" containerName="main" Sep 17 16:00:00.147494 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:00.147456 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="65465fcf-2950-4260-8efa-7592dba5dd7d" containerName="main" Sep 17 16:00:00.150582 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:00.150562 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9" Sep 17 16:00:00.153133 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:00.153112 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\"" Sep 17 16:00:00.159906 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:00.159889 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9" Sep 17 16:00:00.171173 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:00.171148 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9"] Sep 17 16:00:00.466464 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:00.466438 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9"] Sep 17 16:00:00.468180 ip-10-0-128-15 kubenswrapper[2581]: W0917 16:00:00.468153 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486aca51_92b1_4e3d_a217_872b86d22f92.slice/crio-f300977a07c560201cbcb21fd80e29c2f891bcb6b4ce0a1edaaa7fc12a9c4768 WatchSource:0}: Error finding container f300977a07c560201cbcb21fd80e29c2f891bcb6b4ce0a1edaaa7fc12a9c4768: Status 404 returned error can't find the container with id f300977a07c560201cbcb21fd80e29c2f891bcb6b4ce0a1edaaa7fc12a9c4768 Sep 17 16:00:01.304113 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:01.304082 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9" event={"ID":"486aca51-92b1-4e3d-a217-872b86d22f92","Type":"ContainerStarted","Data":"7e21e4c2cd72986a88e78c5ecb936464200c311f8b95d86c6e99247d2a521e41"} Sep 17 16:00:01.304575 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:01.304132 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9" event={"ID":"486aca51-92b1-4e3d-a217-872b86d22f92","Type":"ContainerStarted","Data":"f300977a07c560201cbcb21fd80e29c2f891bcb6b4ce0a1edaaa7fc12a9c4768"} Sep 17 16:00:01.348303 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:01.348254 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9" podStartSLOduration=1.3482395440000001 podStartE2EDuration="1.348239544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:00:01.345891718 +0000 UTC m=+1788.342128467" watchObservedRunningTime="2026-09-17 16:00:01.348239544 +0000 UTC m=+1788.344476321" Sep 17 16:00:01.449335 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:01.449309 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:02.103087 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.103041 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:02.276611 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.276579 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:02.310484 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.310447 2581 generic.go:356] "Generic (PLEG): container finished" podID="486aca51-92b1-4e3d-a217-872b86d22f92" containerID="7e21e4c2cd72986a88e78c5ecb936464200c311f8b95d86c6e99247d2a521e41" exitCode=0 Sep 17 16:00:02.310870 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.310532 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9" event={"ID":"486aca51-92b1-4e3d-a217-872b86d22f92","Type":"ContainerDied","Data":"7e21e4c2cd72986a88e78c5ecb936464200c311f8b95d86c6e99247d2a521e41"} Sep 17 16:00:02.606703 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.606670 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf"] Sep 17 16:00:02.610016 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.609999 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.616539 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.616510 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ddedf5-openshift-default-dockercfg-6s9jv\"" Sep 17 16:00:02.639634 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.639603 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf"] Sep 17 16:00:02.736828 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.736780 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqp8t\" (UniqueName: \"kubernetes.io/projected/ccd73062-11e4-4c6f-8738-42dd74144d4b-kube-api-access-fqp8t\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.736828 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.736839 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-data\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.737072 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.736908 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-credential-socket\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.737072 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.736951 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-workload-certs\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.737072 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.736974 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-workload-socket\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.737072 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.737036 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-envoy\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.737072 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.737064 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd73062-11e4-4c6f-8738-42dd74144d4b-istiod-ca-cert\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.737238 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.737098 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-podinfo\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.737238 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.737120 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-token\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.808838 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.808810 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:02.838196 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.838151 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd73062-11e4-4c6f-8738-42dd74144d4b-istiod-ca-cert\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.838196 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.838202 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-podinfo\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.838392 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.838233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-token\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.838392 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.838269 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqp8t\" (UniqueName: \"kubernetes.io/projected/ccd73062-11e4-4c6f-8738-42dd74144d4b-kube-api-access-fqp8t\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.838392 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.838315 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-data\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.838392 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.838346 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-credential-socket\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.838392 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.838370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-workload-certs\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.838653 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.838391 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-workload-socket\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.838653 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.838439 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-envoy\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.838822 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.838799 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-workload-certs\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.838883 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.838844 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-workload-socket\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.838883 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.838873 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-data\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.839004 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.838985 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-credential-socket\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.839053 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.839019 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd73062-11e4-4c6f-8738-42dd74144d4b-istiod-ca-cert\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.840668 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.840650 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-envoy\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.840813 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.840797 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-podinfo\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.851534 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.851510 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-token\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.851632 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.851546 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqp8t\" (UniqueName: \"kubernetes.io/projected/ccd73062-11e4-4c6f-8738-42dd74144d4b-kube-api-access-fqp8t\") pod \"e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:02.920532 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:02.920440 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:03.203929 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:03.203821 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:03.227485 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:03.227451 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf"] Sep 17 16:00:03.228943 ip-10-0-128-15 kubenswrapper[2581]: W0917 16:00:03.228899 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd73062_11e4_4c6f_8738_42dd74144d4b.slice/crio-cb703c22e43b88ac8568e591c81aa18807147ac260e77c1fcc6854ae3997cb70 WatchSource:0}: Error finding container cb703c22e43b88ac8568e591c81aa18807147ac260e77c1fcc6854ae3997cb70: Status 404 returned error can't find the container with id cb703c22e43b88ac8568e591c81aa18807147ac260e77c1fcc6854ae3997cb70 Sep 17 16:00:03.231230 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:03.231201 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 16:00:03.231334 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:03.231272 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 16:00:03.231334 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:03.231315 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 16:00:03.333076 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:03.333037 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" event={"ID":"ccd73062-11e4-4c6f-8738-42dd74144d4b","Type":"ContainerStarted","Data":"f90b27a9d30dde62292907ccfe817e146e46acba343a683d52874e63befd43f8"} Sep 17 16:00:03.333589 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:03.333136 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" event={"ID":"ccd73062-11e4-4c6f-8738-42dd74144d4b","Type":"ContainerStarted","Data":"cb703c22e43b88ac8568e591c81aa18807147ac260e77c1fcc6854ae3997cb70"} Sep 17 16:00:03.366027 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:03.365957 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" podStartSLOduration=1.365900497 podStartE2EDuration="1.365900497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:00:03.362688186 +0000 UTC m=+1790.358924932" watchObservedRunningTime="2026-09-17 16:00:03.365900497 +0000 UTC m=+1790.362137249" Sep 17 16:00:03.454009 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:03.453930 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9" Sep 17 16:00:03.805994 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:03.805965 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:03.920872 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:03.920841 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:04.339482 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:04.339446 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9" Sep 17 16:00:04.339976 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:04.339487 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9" event={"ID":"486aca51-92b1-4e3d-a217-872b86d22f92","Type":"ContainerDied","Data":"f300977a07c560201cbcb21fd80e29c2f891bcb6b4ce0a1edaaa7fc12a9c4768"} Sep 17 16:00:04.339976 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:04.339530 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f300977a07c560201cbcb21fd80e29c2f891bcb6b4ce0a1edaaa7fc12a9c4768" Sep 17 16:00:04.926659 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:04.926624 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:05.344733 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:05.344701 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:05.345564 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:05.345544 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:05.402854 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:05.402822 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:05.724772 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:05.724692 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:06.403016 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:06.402984 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:07.132887 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:07.132855 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:08.184489 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:08.184454 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:08.295749 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:08.295715 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:08.571408 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:08.571380 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:08.770190 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:08.770156 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:08.893252 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:08.893172 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:09.304742 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:09.304712 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:10.246037 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:10.246001 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:14.127318 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:14.127232 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:21.387119 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:21.387069 2581 patch_prober.go:28] interesting pod/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 17 16:00:21.387497 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:21.387152 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 17 16:00:22.446082 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:22.446046 2581 generic.go:356] "Generic (PLEG): container finished" podID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerID="f90b27a9d30dde62292907ccfe817e146e46acba343a683d52874e63befd43f8" exitCode=0 Sep 17 16:00:22.446605 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:22.446124 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" event={"ID":"ccd73062-11e4-4c6f-8738-42dd74144d4b","Type":"ContainerDied","Data":"f90b27a9d30dde62292907ccfe817e146e46acba343a683d52874e63befd43f8"} Sep 17 16:00:22.446605 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:22.446582 2581 scope.go:117] "RemoveContainer" containerID="f90b27a9d30dde62292907ccfe817e146e46acba343a683d52874e63befd43f8" Sep 17 16:00:22.447640 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:22.447616 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 16:00:22.447700 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:22.447688 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 16:00:22.447738 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:22.447718 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 16:00:23.453460 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:23.453427 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" event={"ID":"ccd73062-11e4-4c6f-8738-42dd74144d4b","Type":"ContainerStarted","Data":"d2c2acd9a5a1d92a2b5f0719841cc359873e2763a8bbef5bf6461942a3cf5880"} Sep 17 16:00:23.921386 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:23.921353 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:23.921623 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:23.921600 2581 patch_prober.go:28] interesting pod/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 17 16:00:23.921707 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:23.921669 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 17 16:00:24.921118 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:24.921077 2581 patch_prober.go:28] interesting pod/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 17 16:00:24.921514 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:24.921140 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 17 16:00:25.921007 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:25.920971 2581 patch_prober.go:28] interesting pod/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 17 16:00:25.921193 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:25.921042 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 17 16:00:26.921865 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:26.921824 2581 patch_prober.go:28] interesting pod/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 17 16:00:26.922277 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:26.921899 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 17 16:00:27.921603 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:27.921565 2581 patch_prober.go:28] interesting pod/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Sep 17 16:00:27.921786 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:27.921632 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Sep 17 16:00:28.925292 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:28.925258 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:28.925748 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:28.925638 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:28.926333 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:28.926311 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:00:30.582581 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:30.582548 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:31.380101 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:31.380067 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:32.471188 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:32.471150 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:33.641601 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:33.641566 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:34.719085 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:34.719037 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:36.285015 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:36.284981 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:36.887204 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:36.887169 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:37.189337 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:37.189243 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:37.781731 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:37.781703 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:42.135255 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:42.135227 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:00:50.182024 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:00:50.181991 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:07.045828 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:07.045791 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:13.179587 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.179552 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:13.560894 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.560852 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln"] Sep 17 16:01:13.561298 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.561279 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="486aca51-92b1-4e3d-a217-872b86d22f92" containerName="cleanup" Sep 17 16:01:13.561389 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.561301 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="486aca51-92b1-4e3d-a217-872b86d22f92" containerName="cleanup" Sep 17 16:01:13.561445 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.561422 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="486aca51-92b1-4e3d-a217-872b86d22f92" containerName="cleanup" Sep 17 16:01:13.563379 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.563359 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.570588 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.570569 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-ec94ed-openshift-default-dockercfg-8zqkw\"" Sep 17 16:01:13.591154 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.591131 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln"] Sep 17 16:01:13.610142 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.610115 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:13.631083 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.631057 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d27v\" (UniqueName: \"kubernetes.io/projected/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-kube-api-access-7d27v\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.631206 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.631092 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-data\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.631206 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.631125 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-workload-socket\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.631206 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.631180 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.631309 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.631235 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-envoy\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.631309 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.631261 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-credential-socket\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.631309 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.631290 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-token\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.631309 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.631305 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-workload-certs\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.631437 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.631323 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-podinfo\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.731746 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.731718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d27v\" (UniqueName: \"kubernetes.io/projected/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-kube-api-access-7d27v\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.731943 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.731754 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-data\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.731943 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.731786 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-workload-socket\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.731943 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.731897 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.732100 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.731998 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-envoy\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.732100 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.732043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-credential-socket\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.732100 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.732085 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-token\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.732262 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.732147 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-data\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.732262 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.732173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-workload-certs\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.732262 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.732216 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-workload-socket\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.732262 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.732221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-podinfo\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.732526 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.732503 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-credential-socket\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.732587 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.732561 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-workload-certs\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.732789 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.732765 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istiod-ca-cert\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.734493 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.734462 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-envoy\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.734656 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.734637 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-podinfo\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.745635 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.745615 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-token\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.745748 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.745697 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d27v\" (UniqueName: \"kubernetes.io/projected/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-kube-api-access-7d27v\") pod \"e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.873705 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.873615 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:13.886349 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:13.886302 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:14.163076 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:14.163048 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln"] Sep 17 16:01:14.164693 ip-10-0-128-15 kubenswrapper[2581]: W0917 16:01:14.164665 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc33bd6f_8abf_4d5c_80d6_cc33ff5f494e.slice/crio-d520cbcf551cbf5724fbdcfec06274eea34abea3dfff4a8ee5b9c327e7d73506 WatchSource:0}: Error finding container d520cbcf551cbf5724fbdcfec06274eea34abea3dfff4a8ee5b9c327e7d73506: Status 404 returned error can't find the container with id d520cbcf551cbf5724fbdcfec06274eea34abea3dfff4a8ee5b9c327e7d73506 Sep 17 16:01:14.166801 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:14.166774 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 16:01:14.166876 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:14.166832 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 16:01:14.167370 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:14.167336 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 16:01:14.728369 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:14.728332 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" event={"ID":"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e","Type":"ContainerStarted","Data":"7c4566238eebc209fcbdcefb2c097959b9738af3c68dcb21319965a7e6489304"} Sep 17 16:01:14.728776 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:14.728418 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" event={"ID":"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e","Type":"ContainerStarted","Data":"d520cbcf551cbf5724fbdcfec06274eea34abea3dfff4a8ee5b9c327e7d73506"} Sep 17 16:01:14.754454 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:14.754408 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" podStartSLOduration=1.7543939160000002 podStartE2EDuration="1.754393916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:14.752244283 +0000 UTC m=+1861.748481029" watchObservedRunningTime="2026-09-17 16:01:14.754393916 +0000 UTC m=+1861.750630666" Sep 17 16:01:14.874026 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:14.873967 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:15.879476 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:15.879443 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:16.234421 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:16.234340 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:16.739298 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:16.739268 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:16.740264 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:16.740241 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:16.855205 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:16.855144 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:17.005260 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:17.005173 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:17.561775 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:17.561746 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:19.178659 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:19.178629 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:20.230826 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:20.230796 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:21.328571 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:21.328539 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:22.291467 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:22.291437 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:22.392184 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:22.392154 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:22.489954 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:22.489906 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:22.791284 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:22.791250 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:22.903272 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:22.903242 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:23.023954 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.023909 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6"] Sep 17 16:01:23.026234 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.026218 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" Sep 17 16:01:23.029783 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.029759 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-ec94ed-dockercfg-trq72\"" Sep 17 16:01:23.030458 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.030441 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ec94ed\"" Sep 17 16:01:23.045793 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.045741 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6"] Sep 17 16:01:23.114193 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.114154 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6\" (UID: \"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" Sep 17 16:01:23.114417 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.114226 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6548\" (UniqueName: \"kubernetes.io/projected/4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf-kube-api-access-g6548\") pod \"payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6\" (UID: \"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" Sep 17 16:01:23.215254 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.215218 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6548\" (UniqueName: \"kubernetes.io/projected/4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf-kube-api-access-g6548\") pod \"payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6\" (UID: \"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" Sep 17 16:01:23.215443 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.215395 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6\" (UID: \"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" Sep 17 16:01:23.216096 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.216077 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6\" (UID: \"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" Sep 17 16:01:23.225140 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.225116 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6548\" (UniqueName: \"kubernetes.io/projected/4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf-kube-api-access-g6548\") pod \"payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6\" (UID: \"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" Sep 17 16:01:23.337211 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.337145 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" Sep 17 16:01:23.616535 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.616496 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6"] Sep 17 16:01:23.617838 ip-10-0-128-15 kubenswrapper[2581]: W0917 16:01:23.617816 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7998e2_2185_4ec5_9faf_eaca8ddcbcbf.slice/crio-d241e69eb928d32bb03e90c5b5ae68f14799b1378274c466342062148b92855e WatchSource:0}: Error finding container d241e69eb928d32bb03e90c5b5ae68f14799b1378274c466342062148b92855e: Status 404 returned error can't find the container with id d241e69eb928d32bb03e90c5b5ae68f14799b1378274c466342062148b92855e Sep 17 16:01:23.780384 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.780337 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" event={"ID":"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf","Type":"ContainerStarted","Data":"00e2a816de460924899e422c14713fa8737c1607e213c342dc45913268f154a3"} Sep 17 16:01:23.780553 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.780415 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" Sep 17 16:01:23.780553 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.780439 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" event={"ID":"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf","Type":"ContainerStarted","Data":"d241e69eb928d32bb03e90c5b5ae68f14799b1378274c466342062148b92855e"} Sep 17 16:01:23.808561 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:23.808516 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" podStartSLOduration=1.808501396 podStartE2EDuration="1.808501396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:23.806039381 +0000 UTC m=+1870.802276127" watchObservedRunningTime="2026-09-17 16:01:23.808501396 +0000 UTC m=+1870.804738141" Sep 17 16:01:34.788032 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:34.787997 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" Sep 17 16:01:37.861940 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:37.861891 2581 generic.go:356] "Generic (PLEG): container finished" podID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerID="7c4566238eebc209fcbdcefb2c097959b9738af3c68dcb21319965a7e6489304" exitCode=0 Sep 17 16:01:37.862469 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:37.861948 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" event={"ID":"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e","Type":"ContainerDied","Data":"7c4566238eebc209fcbdcefb2c097959b9738af3c68dcb21319965a7e6489304"} Sep 17 16:01:37.862469 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:37.862427 2581 scope.go:117] "RemoveContainer" containerID="7c4566238eebc209fcbdcefb2c097959b9738af3c68dcb21319965a7e6489304" Sep 17 16:01:37.863529 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:37.863497 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 16:01:37.863612 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:37.863578 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 16:01:37.863651 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:37.863610 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 17 16:01:38.869358 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:38.869325 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" event={"ID":"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e","Type":"ContainerStarted","Data":"aebb42ebfd0d0a544a5d541459933179fb1110b3e045be17e250c794e1e6d370"} Sep 17 16:01:38.873793 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:38.873767 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:38.874011 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:38.873981 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Sep 17 16:01:38.874094 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:38.874038 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Sep 17 16:01:39.874698 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:39.874656 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Sep 17 16:01:39.875170 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:39.874732 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Sep 17 16:01:40.077907 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:40.077868 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:40.874755 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:40.874711 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Sep 17 16:01:40.875240 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:40.874789 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Sep 17 16:01:40.973126 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:40.973098 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:41.873974 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:41.873903 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Sep 17 16:01:41.874200 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:41.874008 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Sep 17 16:01:42.874134 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:42.874095 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Sep 17 16:01:42.874603 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:42.874178 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Sep 17 16:01:43.874847 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:43.874815 2581 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Sep 17 16:01:43.875249 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:43.874868 2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Sep 17 16:01:44.843860 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:44.843830 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:44.878029 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:44.877999 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:44.878517 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:44.878498 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:44.879188 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:44.879169 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:01:45.844651 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:45.844619 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:46.722298 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:46.722264 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:46.958059 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:46.958023 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:47.526652 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:47.526623 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:48.310625 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:48.310582 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:49.210176 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:49.210145 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:50.034443 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:50.034410 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:50.798757 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:50.798719 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:51.711873 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:51.711841 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:52.582987 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:52.582959 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:01:59.361214 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:59.361174 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6"] Sep 17 16:01:59.361733 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:59.361393 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" podUID="4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf" containerName="payload-pre-processing" containerID="cri-o://00e2a816de460924899e422c14713fa8737c1607e213c342dc45913268f154a3" gracePeriod=30 Sep 17 16:01:59.623430 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:59.623343 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" Sep 17 16:01:59.752787 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:59.752753 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6548\" (UniqueName: \"kubernetes.io/projected/4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf-kube-api-access-g6548\") pod \"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf\" (UID: \"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf\") " Sep 17 16:01:59.752971 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:59.752832 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf-plugins-config-volume\") pod \"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf\" (UID: \"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf\") " Sep 17 16:01:59.753230 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:59.753208 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf" (UID: "4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 16:01:59.754799 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:59.754775 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf-kube-api-access-g6548" (OuterVolumeSpecName: "kube-api-access-g6548") pod "4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf" (UID: "4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf"). InnerVolumeSpecName "kube-api-access-g6548". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:01:59.853543 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:59.853512 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6548\" (UniqueName: \"kubernetes.io/projected/4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf-kube-api-access-g6548\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:01:59.853543 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:01:59.853540 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:00.003200 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:00.003112 2581 generic.go:356] "Generic (PLEG): container finished" podID="4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf" containerID="00e2a816de460924899e422c14713fa8737c1607e213c342dc45913268f154a3" exitCode=0 Sep 17 16:02:00.003200 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:00.003162 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" Sep 17 16:02:00.003420 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:00.003201 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" event={"ID":"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf","Type":"ContainerDied","Data":"00e2a816de460924899e422c14713fa8737c1607e213c342dc45913268f154a3"} Sep 17 16:02:00.003420 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:00.003291 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6" event={"ID":"4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf","Type":"ContainerDied","Data":"d241e69eb928d32bb03e90c5b5ae68f14799b1378274c466342062148b92855e"} Sep 17 16:02:00.003420 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:00.003319 2581 scope.go:117] "RemoveContainer" containerID="00e2a816de460924899e422c14713fa8737c1607e213c342dc45913268f154a3" Sep 17 16:02:00.013338 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:00.013320 2581 scope.go:117] "RemoveContainer" containerID="00e2a816de460924899e422c14713fa8737c1607e213c342dc45913268f154a3" Sep 17 16:02:00.013598 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:02:00.013578 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e2a816de460924899e422c14713fa8737c1607e213c342dc45913268f154a3\": container with ID starting with 00e2a816de460924899e422c14713fa8737c1607e213c342dc45913268f154a3 not found: ID does not exist" containerID="00e2a816de460924899e422c14713fa8737c1607e213c342dc45913268f154a3" Sep 17 16:02:00.013656 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:00.013605 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e2a816de460924899e422c14713fa8737c1607e213c342dc45913268f154a3"} err="failed to get container status \"00e2a816de460924899e422c14713fa8737c1607e213c342dc45913268f154a3\": rpc error: code = NotFound desc = could not find container \"00e2a816de460924899e422c14713fa8737c1607e213c342dc45913268f154a3\": container with ID starting with 00e2a816de460924899e422c14713fa8737c1607e213c342dc45913268f154a3 not found: ID does not exist" Sep 17 16:02:00.028175 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:00.028151 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6"] Sep 17 16:02:00.035024 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:00.035001 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ec94ed-7696c974b-szwl6"] Sep 17 16:02:00.637434 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:00.637397 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:00.867020 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:00.866988 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:01.463068 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:01.463040 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf" path="/var/lib/kubelet/pods/4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf/volumes" Sep 17 16:02:01.905472 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:01.905438 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:02.649747 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:02.649717 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:05.362591 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:05.362555 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:05.467320 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:05.467289 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:06.716070 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:06.716035 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:06.840773 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:06.840740 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:06.970722 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:06.970647 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln"] Sep 17 16:02:06.970888 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:06.970868 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerName="istio-proxy" containerID="cri-o://aebb42ebfd0d0a544a5d541459933179fb1110b3e045be17e250c794e1e6d370" gracePeriod=30 Sep 17 16:02:07.009195 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:07.009163 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:11.003140 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:11.003106 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:11.919662 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:11.919629 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:12.218177 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.218154 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:02:12.250509 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.250486 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-token\") pod \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " Sep 17 16:02:12.250607 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.250534 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istiod-ca-cert\") pod \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " Sep 17 16:02:12.250607 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.250576 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-data\") pod \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " Sep 17 16:02:12.250678 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.250609 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-envoy\") pod \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " Sep 17 16:02:12.250678 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.250643 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d27v\" (UniqueName: \"kubernetes.io/projected/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-kube-api-access-7d27v\") pod \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " Sep 17 16:02:12.250678 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.250670 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-workload-certs\") pod \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " Sep 17 16:02:12.250787 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.250737 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-credential-socket\") pod \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " Sep 17 16:02:12.250787 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.250766 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-podinfo\") pod \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " Sep 17 16:02:12.250887 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.250823 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-workload-socket\") pod \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\" (UID: \"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e\") " Sep 17 16:02:12.251002 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.250933 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" (UID: "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:02:12.251068 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.251034 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" (UID: "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 16:02:12.251164 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.251146 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-data\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:12.251225 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.251170 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istiod-ca-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:12.251277 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.251259 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" (UID: "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:02:12.251468 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.251446 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" (UID: "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:02:12.251468 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.251460 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" (UID: "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:02:12.252712 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.252690 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" (UID: "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:02:12.252841 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.252821 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-kube-api-access-7d27v" (OuterVolumeSpecName: "kube-api-access-7d27v") pod "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" (UID: "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e"). InnerVolumeSpecName "kube-api-access-7d27v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:02:12.252891 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.252844 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" (UID: "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:02:12.253082 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.253055 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" (UID: "fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 16:02:12.351558 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.351528 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-token\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:12.351558 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.351554 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-envoy\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:12.351558 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.351563 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d27v\" (UniqueName: \"kubernetes.io/projected/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-kube-api-access-7d27v\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:12.351766 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.351573 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-workload-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:12.351766 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.351581 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-credential-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:12.351766 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.351590 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-istio-podinfo\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:12.351766 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.351600 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e-workload-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:12.918151 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:12.918118 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:13.082397 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.082362 2581 generic.go:356] "Generic (PLEG): container finished" podID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerID="aebb42ebfd0d0a544a5d541459933179fb1110b3e045be17e250c794e1e6d370" exitCode=0 Sep 17 16:02:13.082577 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.082404 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" Sep 17 16:02:13.082577 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.082444 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" event={"ID":"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e","Type":"ContainerDied","Data":"aebb42ebfd0d0a544a5d541459933179fb1110b3e045be17e250c794e1e6d370"} Sep 17 16:02:13.082577 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.082499 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln" event={"ID":"fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e","Type":"ContainerDied","Data":"d520cbcf551cbf5724fbdcfec06274eea34abea3dfff4a8ee5b9c327e7d73506"} Sep 17 16:02:13.082577 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.082523 2581 scope.go:117] "RemoveContainer" containerID="aebb42ebfd0d0a544a5d541459933179fb1110b3e045be17e250c794e1e6d370" Sep 17 16:02:13.092403 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.092275 2581 scope.go:117] "RemoveContainer" containerID="7c4566238eebc209fcbdcefb2c097959b9738af3c68dcb21319965a7e6489304" Sep 17 16:02:13.100315 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.100298 2581 scope.go:117] "RemoveContainer" containerID="aebb42ebfd0d0a544a5d541459933179fb1110b3e045be17e250c794e1e6d370" Sep 17 16:02:13.100580 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:02:13.100563 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aebb42ebfd0d0a544a5d541459933179fb1110b3e045be17e250c794e1e6d370\": container with ID starting with aebb42ebfd0d0a544a5d541459933179fb1110b3e045be17e250c794e1e6d370 not found: ID does not exist" containerID="aebb42ebfd0d0a544a5d541459933179fb1110b3e045be17e250c794e1e6d370" Sep 17 16:02:13.100633 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.100587 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aebb42ebfd0d0a544a5d541459933179fb1110b3e045be17e250c794e1e6d370"} err="failed to get container status \"aebb42ebfd0d0a544a5d541459933179fb1110b3e045be17e250c794e1e6d370\": rpc error: code = NotFound desc = could not find container \"aebb42ebfd0d0a544a5d541459933179fb1110b3e045be17e250c794e1e6d370\": container with ID starting with aebb42ebfd0d0a544a5d541459933179fb1110b3e045be17e250c794e1e6d370 not found: ID does not exist" Sep 17 16:02:13.100633 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.100603 2581 scope.go:117] "RemoveContainer" containerID="7c4566238eebc209fcbdcefb2c097959b9738af3c68dcb21319965a7e6489304" Sep 17 16:02:13.100820 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:02:13.100804 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4566238eebc209fcbdcefb2c097959b9738af3c68dcb21319965a7e6489304\": container with ID starting with 7c4566238eebc209fcbdcefb2c097959b9738af3c68dcb21319965a7e6489304 not found: ID does not exist" containerID="7c4566238eebc209fcbdcefb2c097959b9738af3c68dcb21319965a7e6489304" Sep 17 16:02:13.100863 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.100824 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4566238eebc209fcbdcefb2c097959b9738af3c68dcb21319965a7e6489304"} err="failed to get container status \"7c4566238eebc209fcbdcefb2c097959b9738af3c68dcb21319965a7e6489304\": rpc error: code = NotFound desc = could not find container \"7c4566238eebc209fcbdcefb2c097959b9738af3c68dcb21319965a7e6489304\": container with ID starting with 7c4566238eebc209fcbdcefb2c097959b9738af3c68dcb21319965a7e6489304 not found: ID does not exist" Sep 17 16:02:13.107469 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.107448 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln"] Sep 17 16:02:13.113632 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.113612 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-ec94ed-openshift-default-5999db448-2c5ln"] Sep 17 16:02:13.462013 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.461978 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" path="/var/lib/kubelet/pods/fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e/volumes" Sep 17 16:02:13.701694 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.701661 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:13.810236 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.810208 2581 scope.go:117] "RemoveContainer" containerID="ba73a027b08a3fa967ebe73ce18df307280df716e58e9e9d0c6189ad245c3d9e" Sep 17 16:02:13.818768 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:13.818745 2581 scope.go:117] "RemoveContainer" containerID="c126beef6e3eb657ca035ccd2da8cd45b1a2423233a3807311be5205185ca5a4" Sep 17 16:02:14.707075 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:14.707046 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:15.638036 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:15.638004 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:26.702697 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:26.702663 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf"] Sep 17 16:02:26.703270 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:26.702945 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" podUID="cfb77b55-a1b3-4416-96cf-a0a7854404e7" containerName="maas-api" containerID="cri-o://9be29c41e3401719c5a2cd7bd02d497291ad548815b15fee7bad23bc0858088a" gracePeriod=30 Sep 17 16:02:26.772400 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:26.772356 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9"] Sep 17 16:02:26.783412 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:26.783385 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-420219-29827680-xj9k9"] Sep 17 16:02:26.797640 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:26.797611 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:26.960727 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:26.960667 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 16:02:27.038032 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.038003 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:27.072785 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.072757 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb77b55-a1b3-4416-96cf-a0a7854404e7-metrics-certs\") pod \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " Sep 17 16:02:27.072955 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.072809 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb77b55-a1b3-4416-96cf-a0a7854404e7-openshift-service-ca\") pod \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " Sep 17 16:02:27.072955 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.072865 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rtl8\" (UniqueName: \"kubernetes.io/projected/cfb77b55-a1b3-4416-96cf-a0a7854404e7-kube-api-access-5rtl8\") pod \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " Sep 17 16:02:27.072955 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.072939 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb77b55-a1b3-4416-96cf-a0a7854404e7-maas-api-tls\") pod \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\" (UID: \"cfb77b55-a1b3-4416-96cf-a0a7854404e7\") " Sep 17 16:02:27.073259 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.073218 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb77b55-a1b3-4416-96cf-a0a7854404e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cfb77b55-a1b3-4416-96cf-a0a7854404e7" (UID: "cfb77b55-a1b3-4416-96cf-a0a7854404e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 16:02:27.074959 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.074926 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb77b55-a1b3-4416-96cf-a0a7854404e7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cfb77b55-a1b3-4416-96cf-a0a7854404e7" (UID: "cfb77b55-a1b3-4416-96cf-a0a7854404e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 16:02:27.075070 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.074974 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb77b55-a1b3-4416-96cf-a0a7854404e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cfb77b55-a1b3-4416-96cf-a0a7854404e7" (UID: "cfb77b55-a1b3-4416-96cf-a0a7854404e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 16:02:27.075149 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.075092 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb77b55-a1b3-4416-96cf-a0a7854404e7-kube-api-access-5rtl8" (OuterVolumeSpecName: "kube-api-access-5rtl8") pod "cfb77b55-a1b3-4416-96cf-a0a7854404e7" (UID: "cfb77b55-a1b3-4416-96cf-a0a7854404e7"). InnerVolumeSpecName "kube-api-access-5rtl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:02:27.162411 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.162373 2581 generic.go:356] "Generic (PLEG): container finished" podID="cfb77b55-a1b3-4416-96cf-a0a7854404e7" containerID="9be29c41e3401719c5a2cd7bd02d497291ad548815b15fee7bad23bc0858088a" exitCode=0 Sep 17 16:02:27.162538 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.162419 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" Sep 17 16:02:27.162538 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.162452 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" event={"ID":"cfb77b55-a1b3-4416-96cf-a0a7854404e7","Type":"ContainerDied","Data":"9be29c41e3401719c5a2cd7bd02d497291ad548815b15fee7bad23bc0858088a"} Sep 17 16:02:27.162538 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.162503 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf" event={"ID":"cfb77b55-a1b3-4416-96cf-a0a7854404e7","Type":"ContainerDied","Data":"709943536b749711caeb6b2184b30e9f4cfe6825990661c775d05515d0d54ac8"} Sep 17 16:02:27.162538 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.162525 2581 scope.go:117] "RemoveContainer" containerID="9be29c41e3401719c5a2cd7bd02d497291ad548815b15fee7bad23bc0858088a" Sep 17 16:02:27.172225 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.172199 2581 scope.go:117] "RemoveContainer" containerID="9be29c41e3401719c5a2cd7bd02d497291ad548815b15fee7bad23bc0858088a" Sep 17 16:02:27.172529 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:02:27.172507 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be29c41e3401719c5a2cd7bd02d497291ad548815b15fee7bad23bc0858088a\": container with ID starting with 9be29c41e3401719c5a2cd7bd02d497291ad548815b15fee7bad23bc0858088a not found: ID does not exist" containerID="9be29c41e3401719c5a2cd7bd02d497291ad548815b15fee7bad23bc0858088a" Sep 17 16:02:27.172596 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.172537 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be29c41e3401719c5a2cd7bd02d497291ad548815b15fee7bad23bc0858088a"} err="failed to get container status \"9be29c41e3401719c5a2cd7bd02d497291ad548815b15fee7bad23bc0858088a\": rpc error: code = NotFound desc = could not find container \"9be29c41e3401719c5a2cd7bd02d497291ad548815b15fee7bad23bc0858088a\": container with ID starting with 9be29c41e3401719c5a2cd7bd02d497291ad548815b15fee7bad23bc0858088a not found: ID does not exist" Sep 17 16:02:27.173879 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.173860 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb77b55-a1b3-4416-96cf-a0a7854404e7-metrics-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:27.173974 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.173881 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb77b55-a1b3-4416-96cf-a0a7854404e7-openshift-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:27.173974 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.173892 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rtl8\" (UniqueName: \"kubernetes.io/projected/cfb77b55-a1b3-4416-96cf-a0a7854404e7-kube-api-access-5rtl8\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:27.173974 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.173901 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfb77b55-a1b3-4416-96cf-a0a7854404e7-maas-api-tls\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:02:27.220413 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.220324 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf"] Sep 17 16:02:27.230592 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.230559 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-420219-8664847ff7-gvcmf"] Sep 17 16:02:27.348158 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.348128 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:27.462235 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.462196 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="486aca51-92b1-4e3d-a217-872b86d22f92" path="/var/lib/kubelet/pods/486aca51-92b1-4e3d-a217-872b86d22f92/volumes" Sep 17 16:02:27.462658 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.462637 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfb77b55-a1b3-4416-96cf-a0a7854404e7" path="/var/lib/kubelet/pods/cfb77b55-a1b3-4416-96cf-a0a7854404e7/volumes" Sep 17 16:02:27.609961 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.609931 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:27.888605 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:27.888522 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:28.105279 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:28.105248 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:29.442130 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:29.442092 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:30.680523 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:30.680486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:32.158575 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:32.158546 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:02:32.979232 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:02:32.979196 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:09.326610 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:09.326576 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:11.040176 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:11.040141 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:13.849204 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:13.849163 2581 scope.go:117] "RemoveContainer" containerID="54202b1d5abba636d0960452c0869db80b376aa6f02679efc58e05d9a1a1b77e" Sep 17 16:03:13.857515 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:13.857489 2581 scope.go:117] "RemoveContainer" containerID="209f18e27a48bc617c509d789ec968ddccdfc234499737645909b7a6ba453470" Sep 17 16:03:29.249810 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:29.249770 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:29.481020 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:29.480990 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb"] Sep 17 16:03:29.481201 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:29.481181 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" containerID="cri-o://4560bc6bddc9ac35c7e91c82d04b727dfdb3b7576ba1a6028a625464075b4ebc" gracePeriod=30 Sep 17 16:03:29.634909 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:29.634875 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:29.642875 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:29.642849 2581 patch_prober.go:28] interesting pod/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 16:03:29.643010 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:29.642921 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 16:03:29.875617 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:29.875588 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:30.023099 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.023027 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv"] Sep 17 16:03:30.023403 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.023389 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb77b55-a1b3-4416-96cf-a0a7854404e7" containerName="maas-api" Sep 17 16:03:30.023446 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.023405 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb77b55-a1b3-4416-96cf-a0a7854404e7" containerName="maas-api" Sep 17 16:03:30.023446 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.023416 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerName="istio-proxy" Sep 17 16:03:30.023446 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.023421 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerName="istio-proxy" Sep 17 16:03:30.023446 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.023430 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf" containerName="payload-pre-processing" Sep 17 16:03:30.023446 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.023435 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf" containerName="payload-pre-processing" Sep 17 16:03:30.023605 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.023453 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerName="istio-proxy" Sep 17 16:03:30.023605 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.023460 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerName="istio-proxy" Sep 17 16:03:30.023605 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.023512 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d7998e2-2185-4ec5-9faf-eaca8ddcbcbf" containerName="payload-pre-processing" Sep 17 16:03:30.023605 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.023520 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfb77b55-a1b3-4416-96cf-a0a7854404e7" containerName="maas-api" Sep 17 16:03:30.023605 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.023526 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerName="istio-proxy" Sep 17 16:03:30.023605 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.023531 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc33bd6f-8abf-4d5c-80d6-cc33ff5f494e" containerName="istio-proxy" Sep 17 16:03:30.026237 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.026221 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv" Sep 17 16:03:30.030197 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.030177 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 16:03:30.033479 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.033463 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\"" Sep 17 16:03:30.073142 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.073121 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv"] Sep 17 16:03:30.125808 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.125775 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5826c1a-1393-47cf-9ccc-426a8d0ab66c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv\" (UID: \"d5826c1a-1393-47cf-9ccc-426a8d0ab66c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv" Sep 17 16:03:30.226759 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.226723 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5826c1a-1393-47cf-9ccc-426a8d0ab66c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv\" (UID: \"d5826c1a-1393-47cf-9ccc-426a8d0ab66c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv" Sep 17 16:03:30.227367 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.227346 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5826c1a-1393-47cf-9ccc-426a8d0ab66c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv\" (UID: \"d5826c1a-1393-47cf-9ccc-426a8d0ab66c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv" Sep 17 16:03:30.335891 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.335860 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv" Sep 17 16:03:30.371336 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.371307 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:30.623801 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:30.623769 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv"] Sep 17 16:03:30.625851 ip-10-0-128-15 kubenswrapper[2581]: W0917 16:03:30.625827 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5826c1a_1393_47cf_9ccc_426a8d0ab66c.slice/crio-0b10c301c3cdf61ae956571cec39d77ae3aa781a927b46887f65a8c72a21a706 WatchSource:0}: Error finding container 0b10c301c3cdf61ae956571cec39d77ae3aa781a927b46887f65a8c72a21a706: Status 404 returned error can't find the container with id 0b10c301c3cdf61ae956571cec39d77ae3aa781a927b46887f65a8c72a21a706 Sep 17 16:03:31.503062 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:31.502968 2581 generic.go:356] "Generic (PLEG): container finished" podID="d5826c1a-1393-47cf-9ccc-426a8d0ab66c" containerID="e31d5308478c6c7f3373e3ff5339e3507112f4c79eec8d4cdcd8c503e730f1f2" exitCode=0 Sep 17 16:03:31.503062 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:31.503041 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv" event={"ID":"d5826c1a-1393-47cf-9ccc-426a8d0ab66c","Type":"ContainerDied","Data":"e31d5308478c6c7f3373e3ff5339e3507112f4c79eec8d4cdcd8c503e730f1f2"} Sep 17 16:03:31.503439 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:31.503086 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv" event={"ID":"d5826c1a-1393-47cf-9ccc-426a8d0ab66c","Type":"ContainerStarted","Data":"0b10c301c3cdf61ae956571cec39d77ae3aa781a927b46887f65a8c72a21a706"} Sep 17 16:03:31.953052 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:31.953014 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:32.634475 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:32.634450 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv" Sep 17 16:03:32.750571 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:32.750543 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5826c1a-1393-47cf-9ccc-426a8d0ab66c-maas-api-service-ca\") pod \"d5826c1a-1393-47cf-9ccc-426a8d0ab66c\" (UID: \"d5826c1a-1393-47cf-9ccc-426a8d0ab66c\") " Sep 17 16:03:32.750872 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:32.750850 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5826c1a-1393-47cf-9ccc-426a8d0ab66c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5826c1a-1393-47cf-9ccc-426a8d0ab66c" (UID: "d5826c1a-1393-47cf-9ccc-426a8d0ab66c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 16:03:32.852101 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:32.852074 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5826c1a-1393-47cf-9ccc-426a8d0ab66c-maas-api-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:33.149630 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:33.149546 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:33.516277 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:33.516247 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv" Sep 17 16:03:33.516448 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:33.516277 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv" event={"ID":"d5826c1a-1393-47cf-9ccc-426a8d0ab66c","Type":"ContainerDied","Data":"0b10c301c3cdf61ae956571cec39d77ae3aa781a927b46887f65a8c72a21a706"} Sep 17 16:03:33.516448 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:33.516326 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b10c301c3cdf61ae956571cec39d77ae3aa781a927b46887f65a8c72a21a706" Sep 17 16:03:34.651369 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.651340 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:34.732957 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.732930 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 16:03:34.870843 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.870809 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-token\") pod \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " Sep 17 16:03:34.871061 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.870854 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-credential-socket\") pod \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " Sep 17 16:03:34.871061 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.870891 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-podinfo\") pod \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " Sep 17 16:03:34.871061 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.870940 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-workload-certs\") pod \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " Sep 17 16:03:34.871061 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.870958 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9s45\" (UniqueName: \"kubernetes.io/projected/cfb89bed-f577-4a9d-bde2-0e388ab3669a-kube-api-access-n9s45\") pod \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " Sep 17 16:03:34.871061 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.870978 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-data\") pod \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " Sep 17 16:03:34.871061 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.871008 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-workload-socket\") pod \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " Sep 17 16:03:34.871061 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.871036 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-envoy\") pod \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " Sep 17 16:03:34.871418 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.871102 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istiod-ca-cert\") pod \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\" (UID: \"cfb89bed-f577-4a9d-bde2-0e388ab3669a\") " Sep 17 16:03:34.871418 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.871261 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cfb89bed-f577-4a9d-bde2-0e388ab3669a" (UID: "cfb89bed-f577-4a9d-bde2-0e388ab3669a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:03:34.871418 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.871278 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cfb89bed-f577-4a9d-bde2-0e388ab3669a" (UID: "cfb89bed-f577-4a9d-bde2-0e388ab3669a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:03:34.871418 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.871300 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-data" (OuterVolumeSpecName: "istio-data") pod "cfb89bed-f577-4a9d-bde2-0e388ab3669a" (UID: "cfb89bed-f577-4a9d-bde2-0e388ab3669a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:03:34.871621 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.871460 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-credential-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:34.871621 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.871479 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-workload-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:34.871621 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.871493 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-data\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:34.871621 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.871489 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cfb89bed-f577-4a9d-bde2-0e388ab3669a" (UID: "cfb89bed-f577-4a9d-bde2-0e388ab3669a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:03:34.871621 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.871558 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cfb89bed-f577-4a9d-bde2-0e388ab3669a" (UID: "cfb89bed-f577-4a9d-bde2-0e388ab3669a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 16:03:34.873124 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.873101 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-token" (OuterVolumeSpecName: "istio-token") pod "cfb89bed-f577-4a9d-bde2-0e388ab3669a" (UID: "cfb89bed-f577-4a9d-bde2-0e388ab3669a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:03:34.873403 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.873386 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cfb89bed-f577-4a9d-bde2-0e388ab3669a" (UID: "cfb89bed-f577-4a9d-bde2-0e388ab3669a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:03:34.873522 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.873503 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cfb89bed-f577-4a9d-bde2-0e388ab3669a" (UID: "cfb89bed-f577-4a9d-bde2-0e388ab3669a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 16:03:34.873569 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.873520 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb89bed-f577-4a9d-bde2-0e388ab3669a-kube-api-access-n9s45" (OuterVolumeSpecName: "kube-api-access-n9s45") pod "cfb89bed-f577-4a9d-bde2-0e388ab3669a" (UID: "cfb89bed-f577-4a9d-bde2-0e388ab3669a"). InnerVolumeSpecName "kube-api-access-n9s45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:03:34.972494 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.972466 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-podinfo\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:34.972494 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.972490 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9s45\" (UniqueName: \"kubernetes.io/projected/cfb89bed-f577-4a9d-bde2-0e388ab3669a-kube-api-access-n9s45\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:34.972494 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.972500 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-workload-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:34.972679 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.972509 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-envoy\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:34.972679 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.972518 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istiod-ca-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:34.972679 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:34.972526 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfb89bed-f577-4a9d-bde2-0e388ab3669a-istio-token\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:35.528465 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:35.528434 2581 generic.go:356] "Generic (PLEG): container finished" podID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerID="4560bc6bddc9ac35c7e91c82d04b727dfdb3b7576ba1a6028a625464075b4ebc" exitCode=0 Sep 17 16:03:35.528616 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:35.528481 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" Sep 17 16:03:35.528616 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:35.528514 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" event={"ID":"cfb89bed-f577-4a9d-bde2-0e388ab3669a","Type":"ContainerDied","Data":"4560bc6bddc9ac35c7e91c82d04b727dfdb3b7576ba1a6028a625464075b4ebc"} Sep 17 16:03:35.528616 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:35.528561 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb" event={"ID":"cfb89bed-f577-4a9d-bde2-0e388ab3669a","Type":"ContainerDied","Data":"0fac493c944fe075edd17ea394c56dc9834058b2d620bc4148ce0d0434bc7291"} Sep 17 16:03:35.528616 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:35.528587 2581 scope.go:117] "RemoveContainer" containerID="4560bc6bddc9ac35c7e91c82d04b727dfdb3b7576ba1a6028a625464075b4ebc" Sep 17 16:03:35.537416 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:35.537401 2581 scope.go:117] "RemoveContainer" containerID="eb4a2079d5f82fb2deddac30723ed861fc1714aa6f2dbe83d36147b161d48bf1" Sep 17 16:03:35.545211 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:35.545186 2581 scope.go:117] "RemoveContainer" containerID="4560bc6bddc9ac35c7e91c82d04b727dfdb3b7576ba1a6028a625464075b4ebc" Sep 17 16:03:35.545487 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:03:35.545467 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4560bc6bddc9ac35c7e91c82d04b727dfdb3b7576ba1a6028a625464075b4ebc\": container with ID starting with 4560bc6bddc9ac35c7e91c82d04b727dfdb3b7576ba1a6028a625464075b4ebc not found: ID does not exist" containerID="4560bc6bddc9ac35c7e91c82d04b727dfdb3b7576ba1a6028a625464075b4ebc" Sep 17 16:03:35.545548 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:35.545493 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4560bc6bddc9ac35c7e91c82d04b727dfdb3b7576ba1a6028a625464075b4ebc"} err="failed to get container status \"4560bc6bddc9ac35c7e91c82d04b727dfdb3b7576ba1a6028a625464075b4ebc\": rpc error: code = NotFound desc = could not find container \"4560bc6bddc9ac35c7e91c82d04b727dfdb3b7576ba1a6028a625464075b4ebc\": container with ID starting with 4560bc6bddc9ac35c7e91c82d04b727dfdb3b7576ba1a6028a625464075b4ebc not found: ID does not exist" Sep 17 16:03:35.545548 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:35.545510 2581 scope.go:117] "RemoveContainer" containerID="eb4a2079d5f82fb2deddac30723ed861fc1714aa6f2dbe83d36147b161d48bf1" Sep 17 16:03:35.545775 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:03:35.545746 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4a2079d5f82fb2deddac30723ed861fc1714aa6f2dbe83d36147b161d48bf1\": container with ID starting with eb4a2079d5f82fb2deddac30723ed861fc1714aa6f2dbe83d36147b161d48bf1 not found: ID does not exist" containerID="eb4a2079d5f82fb2deddac30723ed861fc1714aa6f2dbe83d36147b161d48bf1" Sep 17 16:03:35.545857 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:35.545778 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4a2079d5f82fb2deddac30723ed861fc1714aa6f2dbe83d36147b161d48bf1"} err="failed to get container status \"eb4a2079d5f82fb2deddac30723ed861fc1714aa6f2dbe83d36147b161d48bf1\": rpc error: code = NotFound desc = could not find container \"eb4a2079d5f82fb2deddac30723ed861fc1714aa6f2dbe83d36147b161d48bf1\": container with ID starting with eb4a2079d5f82fb2deddac30723ed861fc1714aa6f2dbe83d36147b161d48bf1 not found: ID does not exist" Sep 17 16:03:35.558956 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:35.558934 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb"] Sep 17 16:03:35.574229 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:35.574204 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-420219-openshift-default-57c96d9858-pz7lb"] Sep 17 16:03:35.945509 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:35.945474 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:37.461798 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:37.461763 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" path="/var/lib/kubelet/pods/cfb89bed-f577-4a9d-bde2-0e388ab3669a/volumes" Sep 17 16:03:41.182841 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:41.182806 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:41.455789 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:41.455688 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:41.697011 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:41.696977 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:41.809455 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:41.809416 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv"] Sep 17 16:03:41.814847 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:41.814818 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddedf5-cfvtv"] Sep 17 16:03:41.842101 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:41.842067 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:42.301460 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:42.301426 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:42.490628 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:42.490586 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:43.461507 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:43.461464 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5826c1a-1393-47cf-9ccc-426a8d0ab66c" path="/var/lib/kubelet/pods/d5826c1a-1393-47cf-9ccc-426a8d0ab66c/volumes" Sep 17 16:03:43.745692 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:43.745615 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:45.170853 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:45.170814 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:50.769892 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:50.769859 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:51.185598 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:51.185561 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf"] Sep 17 16:03:51.185844 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:51.185818 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerName="istio-proxy" containerID="cri-o://d2c2acd9a5a1d92a2b5f0719841cc359873e2763a8bbef5bf6461942a3cf5880" gracePeriod=30 Sep 17 16:03:51.239592 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:51.239563 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:51.697178 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:51.697135 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:52.088633 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:52.088606 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:53.872045 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:53.871998 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:55.422063 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:55.422030 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:56.431556 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.431531 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:03:56.539779 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.539745 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-workload-certs\") pod \"ccd73062-11e4-4c6f-8738-42dd74144d4b\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " Sep 17 16:03:56.539993 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.539785 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-token\") pod \"ccd73062-11e4-4c6f-8738-42dd74144d4b\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " Sep 17 16:03:56.539993 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.539815 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-workload-socket\") pod \"ccd73062-11e4-4c6f-8738-42dd74144d4b\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " Sep 17 16:03:56.539993 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.539846 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-data\") pod \"ccd73062-11e4-4c6f-8738-42dd74144d4b\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " Sep 17 16:03:56.539993 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.539973 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-envoy\") pod \"ccd73062-11e4-4c6f-8738-42dd74144d4b\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " Sep 17 16:03:56.540233 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.540013 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqp8t\" (UniqueName: \"kubernetes.io/projected/ccd73062-11e4-4c6f-8738-42dd74144d4b-kube-api-access-fqp8t\") pod \"ccd73062-11e4-4c6f-8738-42dd74144d4b\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " Sep 17 16:03:56.540233 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.540053 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-podinfo\") pod \"ccd73062-11e4-4c6f-8738-42dd74144d4b\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " Sep 17 16:03:56.540233 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.540085 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ccd73062-11e4-4c6f-8738-42dd74144d4b" (UID: "ccd73062-11e4-4c6f-8738-42dd74144d4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:03:56.540233 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.540099 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd73062-11e4-4c6f-8738-42dd74144d4b-istiod-ca-cert\") pod \"ccd73062-11e4-4c6f-8738-42dd74144d4b\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " Sep 17 16:03:56.540233 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.540136 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-credential-socket\") pod \"ccd73062-11e4-4c6f-8738-42dd74144d4b\" (UID: \"ccd73062-11e4-4c6f-8738-42dd74144d4b\") " Sep 17 16:03:56.540233 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.540162 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-data" (OuterVolumeSpecName: "istio-data") pod "ccd73062-11e4-4c6f-8738-42dd74144d4b" (UID: "ccd73062-11e4-4c6f-8738-42dd74144d4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:03:56.540575 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.540254 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ccd73062-11e4-4c6f-8738-42dd74144d4b" (UID: "ccd73062-11e4-4c6f-8738-42dd74144d4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:03:56.540575 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.540468 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-workload-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:56.540575 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.540474 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ccd73062-11e4-4c6f-8738-42dd74144d4b" (UID: "ccd73062-11e4-4c6f-8738-42dd74144d4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:03:56.540575 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.540488 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-data\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:56.540575 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.540504 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-workload-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:56.540761 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.540583 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd73062-11e4-4c6f-8738-42dd74144d4b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ccd73062-11e4-4c6f-8738-42dd74144d4b" (UID: "ccd73062-11e4-4c6f-8738-42dd74144d4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 16:03:56.542120 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.542096 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ccd73062-11e4-4c6f-8738-42dd74144d4b" (UID: "ccd73062-11e4-4c6f-8738-42dd74144d4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:03:56.542254 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.542172 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ccd73062-11e4-4c6f-8738-42dd74144d4b" (UID: "ccd73062-11e4-4c6f-8738-42dd74144d4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 16:03:56.542482 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.542456 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-token" (OuterVolumeSpecName: "istio-token") pod "ccd73062-11e4-4c6f-8738-42dd74144d4b" (UID: "ccd73062-11e4-4c6f-8738-42dd74144d4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:03:56.542482 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.542465 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd73062-11e4-4c6f-8738-42dd74144d4b-kube-api-access-fqp8t" (OuterVolumeSpecName: "kube-api-access-fqp8t") pod "ccd73062-11e4-4c6f-8738-42dd74144d4b" (UID: "ccd73062-11e4-4c6f-8738-42dd74144d4b"). InnerVolumeSpecName "kube-api-access-fqp8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:03:56.641030 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.640995 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-envoy\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:56.641030 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.641028 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqp8t\" (UniqueName: \"kubernetes.io/projected/ccd73062-11e4-4c6f-8738-42dd74144d4b-kube-api-access-fqp8t\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:56.641030 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.641039 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-podinfo\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:56.641030 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.641048 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccd73062-11e4-4c6f-8738-42dd74144d4b-istiod-ca-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:56.641307 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.641059 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccd73062-11e4-4c6f-8738-42dd74144d4b-credential-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:56.641307 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.641075 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccd73062-11e4-4c6f-8738-42dd74144d4b-istio-token\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:03:56.642441 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.642415 2581 generic.go:356] "Generic (PLEG): container finished" podID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerID="d2c2acd9a5a1d92a2b5f0719841cc359873e2763a8bbef5bf6461942a3cf5880" exitCode=0 Sep 17 16:03:56.642558 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.642466 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" Sep 17 16:03:56.642558 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.642501 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" event={"ID":"ccd73062-11e4-4c6f-8738-42dd74144d4b","Type":"ContainerDied","Data":"d2c2acd9a5a1d92a2b5f0719841cc359873e2763a8bbef5bf6461942a3cf5880"} Sep 17 16:03:56.642558 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.642554 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf" event={"ID":"ccd73062-11e4-4c6f-8738-42dd74144d4b","Type":"ContainerDied","Data":"cb703c22e43b88ac8568e591c81aa18807147ac260e77c1fcc6854ae3997cb70"} Sep 17 16:03:56.642674 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.642577 2581 scope.go:117] "RemoveContainer" containerID="d2c2acd9a5a1d92a2b5f0719841cc359873e2763a8bbef5bf6461942a3cf5880" Sep 17 16:03:56.651718 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.651701 2581 scope.go:117] "RemoveContainer" containerID="f90b27a9d30dde62292907ccfe817e146e46acba343a683d52874e63befd43f8" Sep 17 16:03:56.660381 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.660363 2581 scope.go:117] "RemoveContainer" containerID="d2c2acd9a5a1d92a2b5f0719841cc359873e2763a8bbef5bf6461942a3cf5880" Sep 17 16:03:56.660694 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:03:56.660676 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c2acd9a5a1d92a2b5f0719841cc359873e2763a8bbef5bf6461942a3cf5880\": container with ID starting with d2c2acd9a5a1d92a2b5f0719841cc359873e2763a8bbef5bf6461942a3cf5880 not found: ID does not exist" containerID="d2c2acd9a5a1d92a2b5f0719841cc359873e2763a8bbef5bf6461942a3cf5880" Sep 17 16:03:56.660749 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.660704 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c2acd9a5a1d92a2b5f0719841cc359873e2763a8bbef5bf6461942a3cf5880"} err="failed to get container status \"d2c2acd9a5a1d92a2b5f0719841cc359873e2763a8bbef5bf6461942a3cf5880\": rpc error: code = NotFound desc = could not find container \"d2c2acd9a5a1d92a2b5f0719841cc359873e2763a8bbef5bf6461942a3cf5880\": container with ID starting with d2c2acd9a5a1d92a2b5f0719841cc359873e2763a8bbef5bf6461942a3cf5880 not found: ID does not exist" Sep 17 16:03:56.660749 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.660723 2581 scope.go:117] "RemoveContainer" containerID="f90b27a9d30dde62292907ccfe817e146e46acba343a683d52874e63befd43f8" Sep 17 16:03:56.660989 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:03:56.660969 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90b27a9d30dde62292907ccfe817e146e46acba343a683d52874e63befd43f8\": container with ID starting with f90b27a9d30dde62292907ccfe817e146e46acba343a683d52874e63befd43f8 not found: ID does not exist" containerID="f90b27a9d30dde62292907ccfe817e146e46acba343a683d52874e63befd43f8" Sep 17 16:03:56.661035 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.660995 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90b27a9d30dde62292907ccfe817e146e46acba343a683d52874e63befd43f8"} err="failed to get container status \"f90b27a9d30dde62292907ccfe817e146e46acba343a683d52874e63befd43f8\": rpc error: code = NotFound desc = could not find container \"f90b27a9d30dde62292907ccfe817e146e46acba343a683d52874e63befd43f8\": container with ID starting with f90b27a9d30dde62292907ccfe817e146e46acba343a683d52874e63befd43f8 not found: ID does not exist" Sep 17 16:03:56.694148 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.694109 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf"] Sep 17 16:03:56.708432 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.708396 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ddedf5-openshift-default-65969f687b-m9msf"] Sep 17 16:03:56.904995 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:56.904901 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:03:57.462724 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:03:57.462690 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" path="/var/lib/kubelet/pods/ccd73062-11e4-4c6f-8738-42dd74144d4b/volumes" Sep 17 16:04:02.731378 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:02.731344 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:02.747218 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:02.747189 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk"] Sep 17 16:04:02.747467 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:02.747445 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" podUID="4f7e6cbf-59da-4d79-9623-769a0c5371d2" containerName="payload-processing" containerID="cri-o://0f0f8f3554e35a89afa47f55d96fa31ba9a0b09d53d5319b894397227dce4d5b" gracePeriod=30 Sep 17 16:04:02.767207 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:02.767181 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4"] Sep 17 16:04:02.767406 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:02.767384 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" podUID="1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33" containerName="payload-pre-processing" containerID="cri-o://2d3f8377491b962af286c8b360627f44537d7788795e9cdab4f401dfb5118dd0" gracePeriod=30 Sep 17 16:04:02.885521 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:02.885494 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:03.038063 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.038043 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" Sep 17 16:04:03.040841 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.040820 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" Sep 17 16:04:03.096816 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.096788 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33-plugins-config-volume\") pod \"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33\" (UID: \"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33\") " Sep 17 16:04:03.096985 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.096885 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr5hh\" (UniqueName: \"kubernetes.io/projected/1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33-kube-api-access-mr5hh\") pod \"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33\" (UID: \"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33\") " Sep 17 16:04:03.096985 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.096939 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnzv4\" (UniqueName: \"kubernetes.io/projected/4f7e6cbf-59da-4d79-9623-769a0c5371d2-kube-api-access-vnzv4\") pod \"4f7e6cbf-59da-4d79-9623-769a0c5371d2\" (UID: \"4f7e6cbf-59da-4d79-9623-769a0c5371d2\") " Sep 17 16:04:03.096985 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.096976 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7e6cbf-59da-4d79-9623-769a0c5371d2-plugins-config-volume\") pod \"4f7e6cbf-59da-4d79-9623-769a0c5371d2\" (UID: \"4f7e6cbf-59da-4d79-9623-769a0c5371d2\") " Sep 17 16:04:03.097267 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.097224 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33" (UID: "1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 16:04:03.097416 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.097392 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7e6cbf-59da-4d79-9623-769a0c5371d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f7e6cbf-59da-4d79-9623-769a0c5371d2" (UID: "4f7e6cbf-59da-4d79-9623-769a0c5371d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 16:04:03.099106 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.099076 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33-kube-api-access-mr5hh" (OuterVolumeSpecName: "kube-api-access-mr5hh") pod "1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33" (UID: "1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33"). InnerVolumeSpecName "kube-api-access-mr5hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:04:03.099195 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.099119 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7e6cbf-59da-4d79-9623-769a0c5371d2-kube-api-access-vnzv4" (OuterVolumeSpecName: "kube-api-access-vnzv4") pod "4f7e6cbf-59da-4d79-9623-769a0c5371d2" (UID: "4f7e6cbf-59da-4d79-9623-769a0c5371d2"). InnerVolumeSpecName "kube-api-access-vnzv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:04:03.125409 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.125386 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:03.198023 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.197988 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr5hh\" (UniqueName: \"kubernetes.io/projected/1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33-kube-api-access-mr5hh\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:03.198023 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.198018 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnzv4\" (UniqueName: \"kubernetes.io/projected/4f7e6cbf-59da-4d79-9623-769a0c5371d2-kube-api-access-vnzv4\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:03.198212 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.198031 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f7e6cbf-59da-4d79-9623-769a0c5371d2-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:03.198212 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.198043 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:03.245417 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.245392 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:03.598641 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.598595 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:03.681775 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.681737 2581 generic.go:356] "Generic (PLEG): container finished" podID="1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33" containerID="2d3f8377491b962af286c8b360627f44537d7788795e9cdab4f401dfb5118dd0" exitCode=0 Sep 17 16:04:03.682009 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.681806 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" Sep 17 16:04:03.682009 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.681830 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" event={"ID":"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33","Type":"ContainerDied","Data":"2d3f8377491b962af286c8b360627f44537d7788795e9cdab4f401dfb5118dd0"} Sep 17 16:04:03.682009 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.681888 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4" event={"ID":"1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33","Type":"ContainerDied","Data":"16df0037b746289d36a4cb6867fb2e9d5a03e1c6ff000b3d46d7ba928b13b385"} Sep 17 16:04:03.682009 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.681940 2581 scope.go:117] "RemoveContainer" containerID="2d3f8377491b962af286c8b360627f44537d7788795e9cdab4f401dfb5118dd0" Sep 17 16:04:03.683318 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.683296 2581 generic.go:356] "Generic (PLEG): container finished" podID="4f7e6cbf-59da-4d79-9623-769a0c5371d2" containerID="0f0f8f3554e35a89afa47f55d96fa31ba9a0b09d53d5319b894397227dce4d5b" exitCode=0 Sep 17 16:04:03.683430 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.683362 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" Sep 17 16:04:03.683430 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.683392 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" event={"ID":"4f7e6cbf-59da-4d79-9623-769a0c5371d2","Type":"ContainerDied","Data":"0f0f8f3554e35a89afa47f55d96fa31ba9a0b09d53d5319b894397227dce4d5b"} Sep 17 16:04:03.683430 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.683424 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk" event={"ID":"4f7e6cbf-59da-4d79-9623-769a0c5371d2","Type":"ContainerDied","Data":"0cb6907376455fc3ea6a99bd8cbf2a111abc74dfe3bad6a4f9bbf31de572cddf"} Sep 17 16:04:03.691017 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.690998 2581 scope.go:117] "RemoveContainer" containerID="2d3f8377491b962af286c8b360627f44537d7788795e9cdab4f401dfb5118dd0" Sep 17 16:04:03.691306 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:04:03.691284 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d3f8377491b962af286c8b360627f44537d7788795e9cdab4f401dfb5118dd0\": container with ID starting with 2d3f8377491b962af286c8b360627f44537d7788795e9cdab4f401dfb5118dd0 not found: ID does not exist" containerID="2d3f8377491b962af286c8b360627f44537d7788795e9cdab4f401dfb5118dd0" Sep 17 16:04:03.691384 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.691317 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d3f8377491b962af286c8b360627f44537d7788795e9cdab4f401dfb5118dd0"} err="failed to get container status \"2d3f8377491b962af286c8b360627f44537d7788795e9cdab4f401dfb5118dd0\": rpc error: code = NotFound desc = could not find container \"2d3f8377491b962af286c8b360627f44537d7788795e9cdab4f401dfb5118dd0\": container with ID starting with 2d3f8377491b962af286c8b360627f44537d7788795e9cdab4f401dfb5118dd0 not found: ID does not exist" Sep 17 16:04:03.691384 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.691341 2581 scope.go:117] "RemoveContainer" containerID="0f0f8f3554e35a89afa47f55d96fa31ba9a0b09d53d5319b894397227dce4d5b" Sep 17 16:04:03.699052 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.699035 2581 scope.go:117] "RemoveContainer" containerID="0f0f8f3554e35a89afa47f55d96fa31ba9a0b09d53d5319b894397227dce4d5b" Sep 17 16:04:03.699334 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:04:03.699314 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0f8f3554e35a89afa47f55d96fa31ba9a0b09d53d5319b894397227dce4d5b\": container with ID starting with 0f0f8f3554e35a89afa47f55d96fa31ba9a0b09d53d5319b894397227dce4d5b not found: ID does not exist" containerID="0f0f8f3554e35a89afa47f55d96fa31ba9a0b09d53d5319b894397227dce4d5b" Sep 17 16:04:03.699386 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.699340 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0f8f3554e35a89afa47f55d96fa31ba9a0b09d53d5319b894397227dce4d5b"} err="failed to get container status \"0f0f8f3554e35a89afa47f55d96fa31ba9a0b09d53d5319b894397227dce4d5b\": rpc error: code = NotFound desc = could not find container \"0f0f8f3554e35a89afa47f55d96fa31ba9a0b09d53d5319b894397227dce4d5b\": container with ID starting with 0f0f8f3554e35a89afa47f55d96fa31ba9a0b09d53d5319b894397227dce4d5b not found: ID does not exist" Sep 17 16:04:03.708013 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.707992 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4"] Sep 17 16:04:03.710128 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.710103 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-1d63c0-797f5f5cc4-kzcs4"] Sep 17 16:04:03.723765 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.723714 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk"] Sep 17 16:04:03.731700 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.731680 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1d63c0-688db5c6c7-wdbhk"] Sep 17 16:04:03.797842 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:03.797821 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:04.623976 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:04.623944 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:05.461741 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:05.461712 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33" path="/var/lib/kubelet/pods/1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33/volumes" Sep 17 16:04:05.462127 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:05.462063 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f7e6cbf-59da-4d79-9623-769a0c5371d2" path="/var/lib/kubelet/pods/4f7e6cbf-59da-4d79-9623-769a0c5371d2/volumes" Sep 17 16:04:10.622089 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:10.622056 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:10.720419 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:10.720384 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:10.999377 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:10.999296 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44"] Sep 17 16:04:10.999522 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:10.999502 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" podUID="e58f0775-120c-45da-8eea-9f62402ca713" containerName="istio-proxy" containerID="cri-o://fc8f8a0dd641a42177387b11860fb5c2c30d2a0b6ef205d4beeecaeb759ce765" gracePeriod=30 Sep 17 16:04:11.064381 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.064346 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:11.407969 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.407931 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n"] Sep 17 16:04:11.408319 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408302 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerName="istio-proxy" Sep 17 16:04:11.408412 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408323 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerName="istio-proxy" Sep 17 16:04:11.408412 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408339 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" Sep 17 16:04:11.408412 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408346 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" Sep 17 16:04:11.408412 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408362 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5826c1a-1393-47cf-9ccc-426a8d0ab66c" containerName="revoke-keys" Sep 17 16:04:11.408412 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408371 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5826c1a-1393-47cf-9ccc-426a8d0ab66c" containerName="revoke-keys" Sep 17 16:04:11.408412 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408409 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f7e6cbf-59da-4d79-9623-769a0c5371d2" containerName="payload-processing" Sep 17 16:04:11.408703 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408418 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f7e6cbf-59da-4d79-9623-769a0c5371d2" containerName="payload-processing" Sep 17 16:04:11.408703 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408429 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" Sep 17 16:04:11.408703 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408438 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" Sep 17 16:04:11.408703 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408451 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33" containerName="payload-pre-processing" Sep 17 16:04:11.408703 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408459 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33" containerName="payload-pre-processing" Sep 17 16:04:11.408703 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408541 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerName="istio-proxy" Sep 17 16:04:11.408703 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408552 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" Sep 17 16:04:11.408703 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408561 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfb89bed-f577-4a9d-bde2-0e388ab3669a" containerName="istio-proxy" Sep 17 16:04:11.408703 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408572 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f7e6cbf-59da-4d79-9623-769a0c5371d2" containerName="payload-processing" Sep 17 16:04:11.408703 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408581 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc172b4-e91f-4cbc-bcd0-d0da7e3acd33" containerName="payload-pre-processing" Sep 17 16:04:11.408703 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.408590 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5826c1a-1393-47cf-9ccc-426a8d0ab66c" containerName="revoke-keys" Sep 17 16:04:11.412721 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.412701 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n" Sep 17 16:04:11.415474 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.415455 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 16:04:11.415639 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.415622 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\"" Sep 17 16:04:11.427558 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.427537 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n"] Sep 17 16:04:11.471360 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.471331 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c5c190-004b-42c1-810a-2f8a9feec4ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n\" (UID: \"02c5c190-004b-42c1-810a-2f8a9feec4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n" Sep 17 16:04:11.571875 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.571843 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c5c190-004b-42c1-810a-2f8a9feec4ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n\" (UID: \"02c5c190-004b-42c1-810a-2f8a9feec4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n" Sep 17 16:04:11.572480 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.572460 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c5c190-004b-42c1-810a-2f8a9feec4ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n\" (UID: \"02c5c190-004b-42c1-810a-2f8a9feec4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n" Sep 17 16:04:11.722012 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:11.721927 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n" Sep 17 16:04:12.013203 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:12.013123 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n"] Sep 17 16:04:12.016133 ip-10-0-128-15 kubenswrapper[2581]: W0917 16:04:12.016095 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c5c190_004b_42c1_810a_2f8a9feec4ed.slice/crio-594e9fd9ecf9dde214353063983991ea3f2987b7b2180fbb7f7743e96383a186 WatchSource:0}: Error finding container 594e9fd9ecf9dde214353063983991ea3f2987b7b2180fbb7f7743e96383a186: Status 404 returned error can't find the container with id 594e9fd9ecf9dde214353063983991ea3f2987b7b2180fbb7f7743e96383a186 Sep 17 16:04:12.412592 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:12.412562 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:12.731367 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:12.731332 2581 generic.go:356] "Generic (PLEG): container finished" podID="02c5c190-004b-42c1-810a-2f8a9feec4ed" containerID="49ef1501109cac73001bf03db1bfd457cbf4a5fe103e37a0cca63fb5f069fb8b" exitCode=0 Sep 17 16:04:12.731716 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:12.731372 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n" event={"ID":"02c5c190-004b-42c1-810a-2f8a9feec4ed","Type":"ContainerDied","Data":"49ef1501109cac73001bf03db1bfd457cbf4a5fe103e37a0cca63fb5f069fb8b"} Sep 17 16:04:12.731716 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:12.731424 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n" event={"ID":"02c5c190-004b-42c1-810a-2f8a9feec4ed","Type":"ContainerStarted","Data":"594e9fd9ecf9dde214353063983991ea3f2987b7b2180fbb7f7743e96383a186"} Sep 17 16:04:13.556590 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:13.556556 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:13.861785 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:13.861762 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n" Sep 17 16:04:13.993144 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:13.993115 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c5c190-004b-42c1-810a-2f8a9feec4ed-maas-api-service-ca\") pod \"02c5c190-004b-42c1-810a-2f8a9feec4ed\" (UID: \"02c5c190-004b-42c1-810a-2f8a9feec4ed\") " Sep 17 16:04:13.993424 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:13.993402 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c5c190-004b-42c1-810a-2f8a9feec4ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02c5c190-004b-42c1-810a-2f8a9feec4ed" (UID: "02c5c190-004b-42c1-810a-2f8a9feec4ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 16:04:14.093881 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:14.093810 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c5c190-004b-42c1-810a-2f8a9feec4ed-maas-api-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:14.742484 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:14.742450 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n" event={"ID":"02c5c190-004b-42c1-810a-2f8a9feec4ed","Type":"ContainerDied","Data":"594e9fd9ecf9dde214353063983991ea3f2987b7b2180fbb7f7743e96383a186"} Sep 17 16:04:14.742665 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:14.742498 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="594e9fd9ecf9dde214353063983991ea3f2987b7b2180fbb7f7743e96383a186" Sep 17 16:04:14.742665 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:14.742458 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n" Sep 17 16:04:15.652253 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:15.652212 2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 16:04:15.652641 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:15.652288 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" podUID="e58f0775-120c-45da-8eea-9f62402ca713" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 16:04:16.246734 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.246704 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 16:04:16.312314 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312285 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-istio-envoy\") pod \"e58f0775-120c-45da-8eea-9f62402ca713\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " Sep 17 16:04:16.312465 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312322 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2crt\" (UniqueName: \"kubernetes.io/projected/e58f0775-120c-45da-8eea-9f62402ca713-kube-api-access-l2crt\") pod \"e58f0775-120c-45da-8eea-9f62402ca713\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " Sep 17 16:04:16.312465 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312368 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-workload-socket\") pod \"e58f0775-120c-45da-8eea-9f62402ca713\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " Sep 17 16:04:16.312465 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312400 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58f0775-120c-45da-8eea-9f62402ca713-istiod-ca-cert\") pod \"e58f0775-120c-45da-8eea-9f62402ca713\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " Sep 17 16:04:16.312465 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312415 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-workload-certs\") pod \"e58f0775-120c-45da-8eea-9f62402ca713\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " Sep 17 16:04:16.312465 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312452 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-istio-data\") pod \"e58f0775-120c-45da-8eea-9f62402ca713\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " Sep 17 16:04:16.312735 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312499 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-credential-socket\") pod \"e58f0775-120c-45da-8eea-9f62402ca713\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " Sep 17 16:04:16.312735 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312539 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58f0775-120c-45da-8eea-9f62402ca713-istio-token\") pod \"e58f0775-120c-45da-8eea-9f62402ca713\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " Sep 17 16:04:16.312735 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312562 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58f0775-120c-45da-8eea-9f62402ca713-istio-podinfo\") pod \"e58f0775-120c-45da-8eea-9f62402ca713\" (UID: \"e58f0775-120c-45da-8eea-9f62402ca713\") " Sep 17 16:04:16.312735 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312673 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e58f0775-120c-45da-8eea-9f62402ca713" (UID: "e58f0775-120c-45da-8eea-9f62402ca713"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:04:16.312961 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312728 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e58f0775-120c-45da-8eea-9f62402ca713" (UID: "e58f0775-120c-45da-8eea-9f62402ca713"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:04:16.312961 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312817 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e58f0775-120c-45da-8eea-9f62402ca713" (UID: "e58f0775-120c-45da-8eea-9f62402ca713"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:04:16.312961 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312843 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58f0775-120c-45da-8eea-9f62402ca713-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e58f0775-120c-45da-8eea-9f62402ca713" (UID: "e58f0775-120c-45da-8eea-9f62402ca713"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 16:04:16.312961 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312875 2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-workload-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:16.312961 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.312894 2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-workload-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:16.313166 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.313039 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-istio-data" (OuterVolumeSpecName: "istio-data") pod "e58f0775-120c-45da-8eea-9f62402ca713" (UID: "e58f0775-120c-45da-8eea-9f62402ca713"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:04:16.314443 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.314425 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e58f0775-120c-45da-8eea-9f62402ca713" (UID: "e58f0775-120c-45da-8eea-9f62402ca713"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:04:16.314740 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.314724 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e58f0775-120c-45da-8eea-9f62402ca713-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e58f0775-120c-45da-8eea-9f62402ca713" (UID: "e58f0775-120c-45da-8eea-9f62402ca713"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 17 16:04:16.314906 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.314885 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58f0775-120c-45da-8eea-9f62402ca713-istio-token" (OuterVolumeSpecName: "istio-token") pod "e58f0775-120c-45da-8eea-9f62402ca713" (UID: "e58f0775-120c-45da-8eea-9f62402ca713"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:04:16.315024 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.314933 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58f0775-120c-45da-8eea-9f62402ca713-kube-api-access-l2crt" (OuterVolumeSpecName: "kube-api-access-l2crt") pod "e58f0775-120c-45da-8eea-9f62402ca713" (UID: "e58f0775-120c-45da-8eea-9f62402ca713"). InnerVolumeSpecName "kube-api-access-l2crt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:04:16.413395 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.413368 2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-credential-socket\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:16.413395 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.413393 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58f0775-120c-45da-8eea-9f62402ca713-istio-token\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:16.413557 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.413403 2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58f0775-120c-45da-8eea-9f62402ca713-istio-podinfo\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:16.413557 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.413413 2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-istio-envoy\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:16.413557 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.413421 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2crt\" (UniqueName: \"kubernetes.io/projected/e58f0775-120c-45da-8eea-9f62402ca713-kube-api-access-l2crt\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:16.413557 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.413429 2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58f0775-120c-45da-8eea-9f62402ca713-istiod-ca-cert\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:16.413557 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.413437 2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58f0775-120c-45da-8eea-9f62402ca713-istio-data\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:16.753889 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.753794 2581 generic.go:356] "Generic (PLEG): container finished" podID="e58f0775-120c-45da-8eea-9f62402ca713" containerID="fc8f8a0dd641a42177387b11860fb5c2c30d2a0b6ef205d4beeecaeb759ce765" exitCode=0 Sep 17 16:04:16.753889 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.753836 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" Sep 17 16:04:16.754409 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.753881 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" event={"ID":"e58f0775-120c-45da-8eea-9f62402ca713","Type":"ContainerDied","Data":"fc8f8a0dd641a42177387b11860fb5c2c30d2a0b6ef205d4beeecaeb759ce765"} Sep 17 16:04:16.754409 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.753967 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44" event={"ID":"e58f0775-120c-45da-8eea-9f62402ca713","Type":"ContainerDied","Data":"bf97190dace79a1f6859d92d64436b20c0b8a51310cd6202b6e21dee2ecc1196"} Sep 17 16:04:16.754409 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.754001 2581 scope.go:117] "RemoveContainer" containerID="fc8f8a0dd641a42177387b11860fb5c2c30d2a0b6ef205d4beeecaeb759ce765" Sep 17 16:04:16.763508 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.763492 2581 scope.go:117] "RemoveContainer" containerID="4ccb41635b24fe94727e98d84a2bcd948129a08d80b9ee01023e5c7f8926ff56" Sep 17 16:04:16.771855 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.771837 2581 scope.go:117] "RemoveContainer" containerID="fc8f8a0dd641a42177387b11860fb5c2c30d2a0b6ef205d4beeecaeb759ce765" Sep 17 16:04:16.772115 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:04:16.772096 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8f8a0dd641a42177387b11860fb5c2c30d2a0b6ef205d4beeecaeb759ce765\": container with ID starting with fc8f8a0dd641a42177387b11860fb5c2c30d2a0b6ef205d4beeecaeb759ce765 not found: ID does not exist" containerID="fc8f8a0dd641a42177387b11860fb5c2c30d2a0b6ef205d4beeecaeb759ce765" Sep 17 16:04:16.772202 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.772127 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8f8a0dd641a42177387b11860fb5c2c30d2a0b6ef205d4beeecaeb759ce765"} err="failed to get container status \"fc8f8a0dd641a42177387b11860fb5c2c30d2a0b6ef205d4beeecaeb759ce765\": rpc error: code = NotFound desc = could not find container \"fc8f8a0dd641a42177387b11860fb5c2c30d2a0b6ef205d4beeecaeb759ce765\": container with ID starting with fc8f8a0dd641a42177387b11860fb5c2c30d2a0b6ef205d4beeecaeb759ce765 not found: ID does not exist" Sep 17 16:04:16.772202 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.772150 2581 scope.go:117] "RemoveContainer" containerID="4ccb41635b24fe94727e98d84a2bcd948129a08d80b9ee01023e5c7f8926ff56" Sep 17 16:04:16.772440 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:04:16.772414 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccb41635b24fe94727e98d84a2bcd948129a08d80b9ee01023e5c7f8926ff56\": container with ID starting with 4ccb41635b24fe94727e98d84a2bcd948129a08d80b9ee01023e5c7f8926ff56 not found: ID does not exist" containerID="4ccb41635b24fe94727e98d84a2bcd948129a08d80b9ee01023e5c7f8926ff56" Sep 17 16:04:16.772485 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.772445 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccb41635b24fe94727e98d84a2bcd948129a08d80b9ee01023e5c7f8926ff56"} err="failed to get container status \"4ccb41635b24fe94727e98d84a2bcd948129a08d80b9ee01023e5c7f8926ff56\": rpc error: code = NotFound desc = could not find container \"4ccb41635b24fe94727e98d84a2bcd948129a08d80b9ee01023e5c7f8926ff56\": container with ID starting with 4ccb41635b24fe94727e98d84a2bcd948129a08d80b9ee01023e5c7f8926ff56 not found: ID does not exist" Sep 17 16:04:16.785974 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.785944 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44"] Sep 17 16:04:16.793616 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.788705 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1d63c0-openshift-default-5cfd596f98-dfj44"] Sep 17 16:04:16.879284 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:16.879232 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:17.461427 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:17.461389 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e58f0775-120c-45da-8eea-9f62402ca713" path="/var/lib/kubelet/pods/e58f0775-120c-45da-8eea-9f62402ca713/volumes" Sep 17 16:04:21.669736 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:21.669697 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:22.384371 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.384341 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:22.419933 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.419893 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6"] Sep 17 16:04:22.420124 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.420102 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" podUID="4c28f42c-b413-4a42-98e6-8759f7f4322b" containerName="payload-pre-processing" containerID="cri-o://5d46ed1c8c29204c019e2387f0eb583dec97bb3e7fb812326a1731324cb7a86b" gracePeriod=30 Sep 17 16:04:22.486009 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.485982 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:22.591288 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.591263 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:22.661645 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.661615 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n"] Sep 17 16:04:22.667749 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.667721 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" Sep 17 16:04:22.674416 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.674394 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5818c9-5wb7n"] Sep 17 16:04:22.700449 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.700338 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:22.762959 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.762905 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c28f42c-b413-4a42-98e6-8759f7f4322b-plugins-config-volume\") pod \"4c28f42c-b413-4a42-98e6-8759f7f4322b\" (UID: \"4c28f42c-b413-4a42-98e6-8759f7f4322b\") " Sep 17 16:04:22.763146 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.763047 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlh5s\" (UniqueName: \"kubernetes.io/projected/4c28f42c-b413-4a42-98e6-8759f7f4322b-kube-api-access-dlh5s\") pod \"4c28f42c-b413-4a42-98e6-8759f7f4322b\" (UID: \"4c28f42c-b413-4a42-98e6-8759f7f4322b\") " Sep 17 16:04:22.763325 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.763301 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c28f42c-b413-4a42-98e6-8759f7f4322b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c28f42c-b413-4a42-98e6-8759f7f4322b" (UID: "4c28f42c-b413-4a42-98e6-8759f7f4322b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 16:04:22.764977 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.764949 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c28f42c-b413-4a42-98e6-8759f7f4322b-kube-api-access-dlh5s" (OuterVolumeSpecName: "kube-api-access-dlh5s") pod "4c28f42c-b413-4a42-98e6-8759f7f4322b" (UID: "4c28f42c-b413-4a42-98e6-8759f7f4322b"). InnerVolumeSpecName "kube-api-access-dlh5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:04:22.786737 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.786707 2581 generic.go:356] "Generic (PLEG): container finished" podID="4c28f42c-b413-4a42-98e6-8759f7f4322b" containerID="5d46ed1c8c29204c019e2387f0eb583dec97bb3e7fb812326a1731324cb7a86b" exitCode=0 Sep 17 16:04:22.786872 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.786766 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" Sep 17 16:04:22.786872 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.786797 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" event={"ID":"4c28f42c-b413-4a42-98e6-8759f7f4322b","Type":"ContainerDied","Data":"5d46ed1c8c29204c019e2387f0eb583dec97bb3e7fb812326a1731324cb7a86b"} Sep 17 16:04:22.787022 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.786887 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6" event={"ID":"4c28f42c-b413-4a42-98e6-8759f7f4322b","Type":"ContainerDied","Data":"9f20ac032c258b473d9f8dbaab06e62128475c184cdf6fb4d96aa047f71cb1f9"} Sep 17 16:04:22.787022 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.786936 2581 scope.go:117] "RemoveContainer" containerID="5d46ed1c8c29204c019e2387f0eb583dec97bb3e7fb812326a1731324cb7a86b" Sep 17 16:04:22.796271 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.796252 2581 scope.go:117] "RemoveContainer" containerID="5d46ed1c8c29204c019e2387f0eb583dec97bb3e7fb812326a1731324cb7a86b" Sep 17 16:04:22.796524 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:04:22.796506 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d46ed1c8c29204c019e2387f0eb583dec97bb3e7fb812326a1731324cb7a86b\": container with ID starting with 5d46ed1c8c29204c019e2387f0eb583dec97bb3e7fb812326a1731324cb7a86b not found: ID does not exist" containerID="5d46ed1c8c29204c019e2387f0eb583dec97bb3e7fb812326a1731324cb7a86b" Sep 17 16:04:22.796571 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.796533 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d46ed1c8c29204c019e2387f0eb583dec97bb3e7fb812326a1731324cb7a86b"} err="failed to get container status \"5d46ed1c8c29204c019e2387f0eb583dec97bb3e7fb812326a1731324cb7a86b\": rpc error: code = NotFound desc = could not find container \"5d46ed1c8c29204c019e2387f0eb583dec97bb3e7fb812326a1731324cb7a86b\": container with ID starting with 5d46ed1c8c29204c019e2387f0eb583dec97bb3e7fb812326a1731324cb7a86b not found: ID does not exist" Sep 17 16:04:22.814401 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.814371 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6"] Sep 17 16:04:22.821089 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.821068 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5818c9-7cb659ddd6-kszc6"] Sep 17 16:04:22.864499 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.864473 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlh5s\" (UniqueName: \"kubernetes.io/projected/4c28f42c-b413-4a42-98e6-8759f7f4322b-kube-api-access-dlh5s\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:22.864499 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:22.864497 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c28f42c-b413-4a42-98e6-8759f7f4322b-plugins-config-volume\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:04:23.117003 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:23.116969 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:23.236053 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:23.236022 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:23.462468 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:23.462390 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02c5c190-004b-42c1-810a-2f8a9feec4ed" path="/var/lib/kubelet/pods/02c5c190-004b-42c1-810a-2f8a9feec4ed/volumes" Sep 17 16:04:23.462744 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:23.462732 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c28f42c-b413-4a42-98e6-8759f7f4322b" path="/var/lib/kubelet/pods/4c28f42c-b413-4a42-98e6-8759f7f4322b/volumes" Sep 17 16:04:30.248274 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:30.248237 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:30.350830 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:30.350791 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:30.520962 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:30.520855 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:41.308931 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:41.308824 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:04:41.388791 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:04:41.388747 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:05:26.427311 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:05:26.427244 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:05:56.418777 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:05:56.418702 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:05:57.967022 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:05:57.966972 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:06:13.928577 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:06:13.928542 2581 scope.go:117] "RemoveContainer" containerID="7e21e4c2cd72986a88e78c5ecb936464200c311f8b95d86c6e99247d2a521e41" Sep 17 16:06:38.130145 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:06:38.130099 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:06:38.308204 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:06:38.308164 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:07:12.766868 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:07:12.766787 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:07:12.862947 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:07:12.862908 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:08:10.577384 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:08:10.577342 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:08:31.118951 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:08:31.118891 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:09:13.128241 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128206 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m"] Sep 17 16:09:13.128706 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128686 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e58f0775-120c-45da-8eea-9f62402ca713" containerName="istio-proxy" Sep 17 16:09:13.128750 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128710 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e58f0775-120c-45da-8eea-9f62402ca713" containerName="istio-proxy" Sep 17 16:09:13.128750 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128729 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c28f42c-b413-4a42-98e6-8759f7f4322b" containerName="payload-pre-processing" Sep 17 16:09:13.128750 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128738 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c28f42c-b413-4a42-98e6-8759f7f4322b" containerName="payload-pre-processing" Sep 17 16:09:13.128848 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128750 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerName="istio-proxy" Sep 17 16:09:13.128848 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128761 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerName="istio-proxy" Sep 17 16:09:13.128848 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128773 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e58f0775-120c-45da-8eea-9f62402ca713" containerName="istio-proxy" Sep 17 16:09:13.128848 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128782 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e58f0775-120c-45da-8eea-9f62402ca713" containerName="istio-proxy" Sep 17 16:09:13.128848 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128802 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c5c190-004b-42c1-810a-2f8a9feec4ed" containerName="revoke-keys" Sep 17 16:09:13.128848 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128811 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c5c190-004b-42c1-810a-2f8a9feec4ed" containerName="revoke-keys" Sep 17 16:09:13.129140 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128883 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e58f0775-120c-45da-8eea-9f62402ca713" containerName="istio-proxy" Sep 17 16:09:13.129140 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128896 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e58f0775-120c-45da-8eea-9f62402ca713" containerName="istio-proxy" Sep 17 16:09:13.129140 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128905 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c28f42c-b413-4a42-98e6-8759f7f4322b" containerName="payload-pre-processing" Sep 17 16:09:13.129140 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128935 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccd73062-11e4-4c6f-8738-42dd74144d4b" containerName="istio-proxy" Sep 17 16:09:13.129140 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.128943 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c5c190-004b-42c1-810a-2f8a9feec4ed" containerName="revoke-keys" Sep 17 16:09:13.132131 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.132110 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.135248 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.135226 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3c8d1b21-kserve-self-signed-certs\"" Sep 17 16:09:13.146810 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.146789 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m"] Sep 17 16:09:13.234635 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.234605 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:09:13.263115 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.263089 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-model-cache\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.263204 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.263143 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-dshm\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.263246 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.263208 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-home\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.263289 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.263276 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-tmp-dir\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.263327 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.263304 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttczb\" (UniqueName: \"kubernetes.io/projected/6a68001e-5241-4ad0-bad8-472bca534222-kube-api-access-ttczb\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.263362 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.263326 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a68001e-5241-4ad0-bad8-472bca534222-tls-certs\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.363862 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.363833 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-tmp-dir\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.364004 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.363869 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttczb\" (UniqueName: \"kubernetes.io/projected/6a68001e-5241-4ad0-bad8-472bca534222-kube-api-access-ttczb\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.364004 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.363888 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a68001e-5241-4ad0-bad8-472bca534222-tls-certs\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.364004 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.363922 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-model-cache\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.364119 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.364024 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-dshm\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.364119 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.364063 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-home\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.364297 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.364281 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-tmp-dir\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.364346 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.364305 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-model-cache\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.364346 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.364314 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-home\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.366123 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.366106 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-dshm\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.375064 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.375046 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3c8d1b21-kserve-self-signed-certs\"" Sep 17 16:09:13.382528 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.382469 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttczb\" (UniqueName: \"kubernetes.io/projected/6a68001e-5241-4ad0-bad8-472bca534222-kube-api-access-ttczb\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.386029 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.386009 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a68001e-5241-4ad0-bad8-472bca534222-tls-certs\") pod \"e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.441495 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.441465 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:13.732960 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:13.732873 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m"] Sep 17 16:09:13.735740 ip-10-0-128-15 kubenswrapper[2581]: W0917 16:09:13.735707 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a68001e_5241_4ad0_bad8_472bca534222.slice/crio-f07d0c7870dcdce5cc7c7781642d9b7ee19ad961db027e19eb81b0713e0cae56 WatchSource:0}: Error finding container f07d0c7870dcdce5cc7c7781642d9b7ee19ad961db027e19eb81b0713e0cae56: Status 404 returned error can't find the container with id f07d0c7870dcdce5cc7c7781642d9b7ee19ad961db027e19eb81b0713e0cae56 Sep 17 16:09:14.203230 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:14.203198 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" event={"ID":"6a68001e-5241-4ad0-bad8-472bca534222","Type":"ContainerStarted","Data":"4c2f18144af894099a43412a67b41c20fa19c6169a3ef18063cecceb4d15db34"} Sep 17 16:09:14.203698 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:14.203237 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" event={"ID":"6a68001e-5241-4ad0-bad8-472bca534222","Type":"ContainerStarted","Data":"f07d0c7870dcdce5cc7c7781642d9b7ee19ad961db027e19eb81b0713e0cae56"} Sep 17 16:09:14.238483 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:14.238431 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" podStartSLOduration=1.238412309 podStartE2EDuration="1.238412309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:09:14.236522167 +0000 UTC m=+2341.232758924" watchObservedRunningTime="2026-09-17 16:09:14.238412309 +0000 UTC m=+2341.234649055" Sep 17 16:09:20.030603 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:20.030569 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:09:23.442128 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:23.442076 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:23.442565 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:23.442149 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:23.457974 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:23.457942 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:24.265661 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:24.265628 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:40.919618 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:40.919581 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m"] Sep 17 16:09:40.942113 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:40.942086 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:09:42.902521 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:42.902486 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:09:55.937863 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:55.937813 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" podUID="6a68001e-5241-4ad0-bad8-472bca534222" containerName="main" containerID="cri-o://4c2f18144af894099a43412a67b41c20fa19c6169a3ef18063cecceb4d15db34" gracePeriod=45 Sep 17 16:09:56.191741 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.191681 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:56.315424 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.315386 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-dshm\") pod \"6a68001e-5241-4ad0-bad8-472bca534222\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " Sep 17 16:09:56.315424 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.315426 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-home\") pod \"6a68001e-5241-4ad0-bad8-472bca534222\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " Sep 17 16:09:56.315683 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.315443 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a68001e-5241-4ad0-bad8-472bca534222-tls-certs\") pod \"6a68001e-5241-4ad0-bad8-472bca534222\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " Sep 17 16:09:56.315683 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.315481 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-model-cache\") pod \"6a68001e-5241-4ad0-bad8-472bca534222\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " Sep 17 16:09:56.315683 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.315519 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttczb\" (UniqueName: \"kubernetes.io/projected/6a68001e-5241-4ad0-bad8-472bca534222-kube-api-access-ttczb\") pod \"6a68001e-5241-4ad0-bad8-472bca534222\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " Sep 17 16:09:56.315683 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.315552 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-tmp-dir\") pod \"6a68001e-5241-4ad0-bad8-472bca534222\" (UID: \"6a68001e-5241-4ad0-bad8-472bca534222\") " Sep 17 16:09:56.315866 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.315722 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-home" (OuterVolumeSpecName: "home") pod "6a68001e-5241-4ad0-bad8-472bca534222" (UID: "6a68001e-5241-4ad0-bad8-472bca534222"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:09:56.315866 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.315772 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a68001e-5241-4ad0-bad8-472bca534222" (UID: "6a68001e-5241-4ad0-bad8-472bca534222"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:09:56.315971 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.315936 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a68001e-5241-4ad0-bad8-472bca534222" (UID: "6a68001e-5241-4ad0-bad8-472bca534222"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:09:56.317720 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.317687 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a68001e-5241-4ad0-bad8-472bca534222-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a68001e-5241-4ad0-bad8-472bca534222" (UID: "6a68001e-5241-4ad0-bad8-472bca534222"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 16:09:56.317832 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.317732 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-dshm" (OuterVolumeSpecName: "dshm") pod "6a68001e-5241-4ad0-bad8-472bca534222" (UID: "6a68001e-5241-4ad0-bad8-472bca534222"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 16:09:56.317832 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.317744 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a68001e-5241-4ad0-bad8-472bca534222-kube-api-access-ttczb" (OuterVolumeSpecName: "kube-api-access-ttczb") pod "6a68001e-5241-4ad0-bad8-472bca534222" (UID: "6a68001e-5241-4ad0-bad8-472bca534222"). InnerVolumeSpecName "kube-api-access-ttczb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:09:56.416183 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.416153 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-model-cache\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:09:56.416183 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.416179 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttczb\" (UniqueName: \"kubernetes.io/projected/6a68001e-5241-4ad0-bad8-472bca534222-kube-api-access-ttczb\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:09:56.416381 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.416192 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-tmp-dir\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:09:56.416381 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.416205 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-dshm\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:09:56.416381 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.416217 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a68001e-5241-4ad0-bad8-472bca534222-home\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:09:56.416381 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.416228 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a68001e-5241-4ad0-bad8-472bca534222-tls-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:09:56.417161 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.417138 2581 generic.go:356] "Generic (PLEG): container finished" podID="6a68001e-5241-4ad0-bad8-472bca534222" containerID="4c2f18144af894099a43412a67b41c20fa19c6169a3ef18063cecceb4d15db34" exitCode=0 Sep 17 16:09:56.417244 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.417200 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" Sep 17 16:09:56.417244 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.417209 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" event={"ID":"6a68001e-5241-4ad0-bad8-472bca534222","Type":"ContainerDied","Data":"4c2f18144af894099a43412a67b41c20fa19c6169a3ef18063cecceb4d15db34"} Sep 17 16:09:56.417244 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.417234 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m" event={"ID":"6a68001e-5241-4ad0-bad8-472bca534222","Type":"ContainerDied","Data":"f07d0c7870dcdce5cc7c7781642d9b7ee19ad961db027e19eb81b0713e0cae56"} Sep 17 16:09:56.417350 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.417251 2581 scope.go:117] "RemoveContainer" containerID="4c2f18144af894099a43412a67b41c20fa19c6169a3ef18063cecceb4d15db34" Sep 17 16:09:56.427081 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.427060 2581 scope.go:117] "RemoveContainer" containerID="4c2f18144af894099a43412a67b41c20fa19c6169a3ef18063cecceb4d15db34" Sep 17 16:09:56.427353 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:09:56.427334 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2f18144af894099a43412a67b41c20fa19c6169a3ef18063cecceb4d15db34\": container with ID starting with 4c2f18144af894099a43412a67b41c20fa19c6169a3ef18063cecceb4d15db34 not found: ID does not exist" containerID="4c2f18144af894099a43412a67b41c20fa19c6169a3ef18063cecceb4d15db34" Sep 17 16:09:56.427427 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.427365 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2f18144af894099a43412a67b41c20fa19c6169a3ef18063cecceb4d15db34"} err="failed to get container status \"4c2f18144af894099a43412a67b41c20fa19c6169a3ef18063cecceb4d15db34\": rpc error: code = NotFound desc = could not find container \"4c2f18144af894099a43412a67b41c20fa19c6169a3ef18063cecceb4d15db34\": container with ID starting with 4c2f18144af894099a43412a67b41c20fa19c6169a3ef18063cecceb4d15db34 not found: ID does not exist" Sep 17 16:09:56.450980 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.450925 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m"] Sep 17 16:09:56.459424 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:56.459406 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3c8d1b21-kserve-7567c5f66c-7pv5m"] Sep 17 16:09:57.461468 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:09:57.461438 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a68001e-5241-4ad0-bad8-472bca534222" path="/var/lib/kubelet/pods/6a68001e-5241-4ad0-bad8-472bca534222/volumes" Sep 17 16:10:02.363467 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.363423 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g"] Sep 17 16:10:02.363978 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.363871 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a68001e-5241-4ad0-bad8-472bca534222" containerName="main" Sep 17 16:10:02.363978 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.363889 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a68001e-5241-4ad0-bad8-472bca534222" containerName="main" Sep 17 16:10:02.364114 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.364002 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a68001e-5241-4ad0-bad8-472bca534222" containerName="main" Sep 17 16:10:02.368242 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.368221 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.371251 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.371231 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 16:10:02.371718 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.371698 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 17 16:10:02.371859 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.371841 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 16:10:02.371968 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.371869 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 16:10:02.371968 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.371938 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mdtj9\"" Sep 17 16:10:02.409415 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.409373 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g"] Sep 17 16:10:02.412564 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.412536 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq"] Sep 17 16:10:02.416190 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.416169 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.434976 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.434949 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq"] Sep 17 16:10:02.466242 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.466212 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-openshift-service-ca\") pod \"maas-api-d5fdc6457-kkw8g\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.466425 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.466266 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5phvb\" (UniqueName: \"kubernetes.io/projected/d85792b7-d7b1-40e7-97b1-9cd8002906d1-kube-api-access-5phvb\") pod \"maas-api-7698c4fdd9-6ghjq\" (UID: \"d85792b7-d7b1-40e7-97b1-9cd8002906d1\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.466425 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.466349 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d85792b7-d7b1-40e7-97b1-9cd8002906d1-metrics-certs\") pod \"maas-api-7698c4fdd9-6ghjq\" (UID: \"d85792b7-d7b1-40e7-97b1-9cd8002906d1\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.466425 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.466378 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jssbw\" (UniqueName: \"kubernetes.io/projected/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-kube-api-access-jssbw\") pod \"maas-api-d5fdc6457-kkw8g\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.466425 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.466416 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-metrics-certs\") pod \"maas-api-d5fdc6457-kkw8g\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.466586 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.466469 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d85792b7-d7b1-40e7-97b1-9cd8002906d1-maas-api-tls\") pod \"maas-api-7698c4fdd9-6ghjq\" (UID: \"d85792b7-d7b1-40e7-97b1-9cd8002906d1\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.466586 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.466497 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d85792b7-d7b1-40e7-97b1-9cd8002906d1-openshift-service-ca\") pod \"maas-api-7698c4fdd9-6ghjq\" (UID: \"d85792b7-d7b1-40e7-97b1-9cd8002906d1\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.466586 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.466515 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-maas-api-tls\") pod \"maas-api-d5fdc6457-kkw8g\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.567171 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.567128 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d85792b7-d7b1-40e7-97b1-9cd8002906d1-metrics-certs\") pod \"maas-api-7698c4fdd9-6ghjq\" (UID: \"d85792b7-d7b1-40e7-97b1-9cd8002906d1\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.567171 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.567174 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jssbw\" (UniqueName: \"kubernetes.io/projected/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-kube-api-access-jssbw\") pod \"maas-api-d5fdc6457-kkw8g\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.567436 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.567298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-metrics-certs\") pod \"maas-api-d5fdc6457-kkw8g\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.567436 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.567364 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d85792b7-d7b1-40e7-97b1-9cd8002906d1-maas-api-tls\") pod \"maas-api-7698c4fdd9-6ghjq\" (UID: \"d85792b7-d7b1-40e7-97b1-9cd8002906d1\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.567436 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.567415 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d85792b7-d7b1-40e7-97b1-9cd8002906d1-openshift-service-ca\") pod \"maas-api-7698c4fdd9-6ghjq\" (UID: \"d85792b7-d7b1-40e7-97b1-9cd8002906d1\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.567626 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.567451 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-maas-api-tls\") pod \"maas-api-d5fdc6457-kkw8g\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.567626 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.567497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-openshift-service-ca\") pod \"maas-api-d5fdc6457-kkw8g\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.567626 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.567585 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5phvb\" (UniqueName: \"kubernetes.io/projected/d85792b7-d7b1-40e7-97b1-9cd8002906d1-kube-api-access-5phvb\") pod \"maas-api-7698c4fdd9-6ghjq\" (UID: \"d85792b7-d7b1-40e7-97b1-9cd8002906d1\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.568210 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.568181 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d85792b7-d7b1-40e7-97b1-9cd8002906d1-openshift-service-ca\") pod \"maas-api-7698c4fdd9-6ghjq\" (UID: \"d85792b7-d7b1-40e7-97b1-9cd8002906d1\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.568366 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.568341 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-openshift-service-ca\") pod \"maas-api-d5fdc6457-kkw8g\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.570317 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.570293 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d85792b7-d7b1-40e7-97b1-9cd8002906d1-maas-api-tls\") pod \"maas-api-7698c4fdd9-6ghjq\" (UID: \"d85792b7-d7b1-40e7-97b1-9cd8002906d1\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.570405 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.570312 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d85792b7-d7b1-40e7-97b1-9cd8002906d1-metrics-certs\") pod \"maas-api-7698c4fdd9-6ghjq\" (UID: \"d85792b7-d7b1-40e7-97b1-9cd8002906d1\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.570451 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.570396 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-maas-api-tls\") pod \"maas-api-d5fdc6457-kkw8g\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.570451 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.570406 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-metrics-certs\") pod \"maas-api-d5fdc6457-kkw8g\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.585616 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.585587 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jssbw\" (UniqueName: \"kubernetes.io/projected/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-kube-api-access-jssbw\") pod \"maas-api-d5fdc6457-kkw8g\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.591677 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.591654 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5phvb\" (UniqueName: \"kubernetes.io/projected/d85792b7-d7b1-40e7-97b1-9cd8002906d1-kube-api-access-5phvb\") pod \"maas-api-7698c4fdd9-6ghjq\" (UID: \"d85792b7-d7b1-40e7-97b1-9cd8002906d1\") " pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.678891 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.678800 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:02.726207 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.726172 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:02.967366 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.967325 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g"] Sep 17 16:10:02.969530 ip-10-0-128-15 kubenswrapper[2581]: W0917 16:10:02.969503 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f81d981_d12a_4541_b0fd_ab2f1e99d6f7.slice/crio-83d974fcd1c339c0e6982f507b9982e882002f5ec5b4cf936b1e96b6a75e9c3f WatchSource:0}: Error finding container 83d974fcd1c339c0e6982f507b9982e882002f5ec5b4cf936b1e96b6a75e9c3f: Status 404 returned error can't find the container with id 83d974fcd1c339c0e6982f507b9982e882002f5ec5b4cf936b1e96b6a75e9c3f Sep 17 16:10:02.971005 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:02.970985 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 16:10:03.007649 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:03.007624 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq"] Sep 17 16:10:03.008478 ip-10-0-128-15 kubenswrapper[2581]: W0917 16:10:03.008452 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85792b7_d7b1_40e7_97b1_9cd8002906d1.slice/crio-d3e34b40db8cc1888975d22b602b61bdf46ae39aee5b0083893f256caad5455e WatchSource:0}: Error finding container d3e34b40db8cc1888975d22b602b61bdf46ae39aee5b0083893f256caad5455e: Status 404 returned error can't find the container with id d3e34b40db8cc1888975d22b602b61bdf46ae39aee5b0083893f256caad5455e Sep 17 16:10:03.455239 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:03.455197 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" event={"ID":"d85792b7-d7b1-40e7-97b1-9cd8002906d1","Type":"ContainerStarted","Data":"d3e34b40db8cc1888975d22b602b61bdf46ae39aee5b0083893f256caad5455e"} Sep 17 16:10:03.461518 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:03.461485 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" event={"ID":"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7","Type":"ContainerStarted","Data":"83d974fcd1c339c0e6982f507b9982e882002f5ec5b4cf936b1e96b6a75e9c3f"} Sep 17 16:10:04.462890 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:04.462856 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" event={"ID":"d85792b7-d7b1-40e7-97b1-9cd8002906d1","Type":"ContainerStarted","Data":"fdcaa101c744593ab741cccbfab1edb8cb291a8e2e3df8c5a764dc09b8627c5a"} Sep 17 16:10:04.463313 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:04.462964 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:04.464365 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:04.464341 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" event={"ID":"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7","Type":"ContainerStarted","Data":"29a6c7a71102a15c21e4d346c40688f220da5d92a6c26d49e3b63e991e7b2ecb"} Sep 17 16:10:04.464487 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:04.464391 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:04.487285 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:04.487232 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" podStartSLOduration=1.7015550130000001 podStartE2EDuration="2.487215033s" totalImagesPullingTime="785.66002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:10:04.486412484 +0000 UTC m=+2391.482649230" watchObservedRunningTime="2026-09-17 16:10:04.487215033 +0000 UTC m=+2391.483451780" Sep 17 16:10:04.518894 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:04.518854 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" podStartSLOduration=1.584184606 podStartE2EDuration="2.518842182s" totalImagesPullingTime="934.657576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:10:04.516237923 +0000 UTC m=+2391.512474670" watchObservedRunningTime="2026-09-17 16:10:04.518842182 +0000 UTC m=+2391.515078928" Sep 17 16:10:10.474222 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:10.474187 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7698c4fdd9-6ghjq" Sep 17 16:10:10.474683 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:10.474580 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:10.579742 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:10.579710 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g"] Sep 17 16:10:10.579986 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:10.579964 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" podUID="9f81d981-d12a-4541-b0fd-ab2f1e99d6f7" containerName="maas-api" containerID="cri-o://29a6c7a71102a15c21e4d346c40688f220da5d92a6c26d49e3b63e991e7b2ecb" gracePeriod=30 Sep 17 16:10:10.821929 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:10.821895 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:10.941862 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:10.941832 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jssbw\" (UniqueName: \"kubernetes.io/projected/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-kube-api-access-jssbw\") pod \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " Sep 17 16:10:10.942098 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:10.941892 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-maas-api-tls\") pod \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " Sep 17 16:10:10.942098 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:10.941961 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-openshift-service-ca\") pod \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " Sep 17 16:10:10.942098 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:10.941999 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-metrics-certs\") pod \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\" (UID: \"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7\") " Sep 17 16:10:10.942332 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:10.942309 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f81d981-d12a-4541-b0fd-ab2f1e99d6f7" (UID: "9f81d981-d12a-4541-b0fd-ab2f1e99d6f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 16:10:10.944031 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:10.944006 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f81d981-d12a-4541-b0fd-ab2f1e99d6f7" (UID: "9f81d981-d12a-4541-b0fd-ab2f1e99d6f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 16:10:10.944128 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:10.944050 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9f81d981-d12a-4541-b0fd-ab2f1e99d6f7" (UID: "9f81d981-d12a-4541-b0fd-ab2f1e99d6f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 16:10:10.944128 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:10.944078 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-kube-api-access-jssbw" (OuterVolumeSpecName: "kube-api-access-jssbw") pod "9f81d981-d12a-4541-b0fd-ab2f1e99d6f7" (UID: "9f81d981-d12a-4541-b0fd-ab2f1e99d6f7"). InnerVolumeSpecName "kube-api-access-jssbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 16:10:11.043464 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:11.043435 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-maas-api-tls\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:10:11.043464 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:11.043463 2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-openshift-service-ca\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:10:11.043641 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:11.043473 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-metrics-certs\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:10:11.043641 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:11.043483 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jssbw\" (UniqueName: \"kubernetes.io/projected/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7-kube-api-access-jssbw\") on node \"ip-10-0-128-15.ec2.internal\" DevicePath \"\"" Sep 17 16:10:11.501371 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:11.501337 2581 generic.go:356] "Generic (PLEG): container finished" podID="9f81d981-d12a-4541-b0fd-ab2f1e99d6f7" containerID="29a6c7a71102a15c21e4d346c40688f220da5d92a6c26d49e3b63e991e7b2ecb" exitCode=0 Sep 17 16:10:11.501787 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:11.501376 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" Sep 17 16:10:11.501787 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:11.501428 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" event={"ID":"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7","Type":"ContainerDied","Data":"29a6c7a71102a15c21e4d346c40688f220da5d92a6c26d49e3b63e991e7b2ecb"} Sep 17 16:10:11.501787 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:11.501473 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g" event={"ID":"9f81d981-d12a-4541-b0fd-ab2f1e99d6f7","Type":"ContainerDied","Data":"83d974fcd1c339c0e6982f507b9982e882002f5ec5b4cf936b1e96b6a75e9c3f"} Sep 17 16:10:11.501787 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:11.501494 2581 scope.go:117] "RemoveContainer" containerID="29a6c7a71102a15c21e4d346c40688f220da5d92a6c26d49e3b63e991e7b2ecb" Sep 17 16:10:11.510762 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:11.510746 2581 scope.go:117] "RemoveContainer" containerID="29a6c7a71102a15c21e4d346c40688f220da5d92a6c26d49e3b63e991e7b2ecb" Sep 17 16:10:11.511060 ip-10-0-128-15 kubenswrapper[2581]: E0917 16:10:11.511043 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a6c7a71102a15c21e4d346c40688f220da5d92a6c26d49e3b63e991e7b2ecb\": container with ID starting with 29a6c7a71102a15c21e4d346c40688f220da5d92a6c26d49e3b63e991e7b2ecb not found: ID does not exist" containerID="29a6c7a71102a15c21e4d346c40688f220da5d92a6c26d49e3b63e991e7b2ecb" Sep 17 16:10:11.511121 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:11.511067 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a6c7a71102a15c21e4d346c40688f220da5d92a6c26d49e3b63e991e7b2ecb"} err="failed to get container status \"29a6c7a71102a15c21e4d346c40688f220da5d92a6c26d49e3b63e991e7b2ecb\": rpc error: code = NotFound desc = could not find container \"29a6c7a71102a15c21e4d346c40688f220da5d92a6c26d49e3b63e991e7b2ecb\": container with ID starting with 29a6c7a71102a15c21e4d346c40688f220da5d92a6c26d49e3b63e991e7b2ecb not found: ID does not exist" Sep 17 16:10:11.519590 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:11.519567 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g"] Sep 17 16:10:11.523609 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:11.523584 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-d5fdc6457-kkw8g"] Sep 17 16:10:13.462325 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:13.462292 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f81d981-d12a-4541-b0fd-ab2f1e99d6f7" path="/var/lib/kubelet/pods/9f81d981-d12a-4541-b0fd-ab2f1e99d6f7/volumes" Sep 17 16:10:13.972724 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:13.972689 2581 scope.go:117] "RemoveContainer" containerID="49ef1501109cac73001bf03db1bfd457cbf4a5fe103e37a0cca63fb5f069fb8b" Sep 17 16:10:13.980610 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:13.980590 2581 scope.go:117] "RemoveContainer" containerID="e31d5308478c6c7f3373e3ff5339e3507112f4c79eec8d4cdcd8c503e730f1f2" Sep 17 16:10:32.526872 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:32.526838 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f85fc86df-hl4bd_3898c87d-f4a6-4ce0-8190-a08ed2ca28e9/authorino/0.log" Sep 17 16:10:39.163643 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:39.163569 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5df9996869-cq6xm_24c16fd2-7c59-480f-a5b1-c52fffe7748d/copy-manifests/0.log" Sep 17 16:10:39.171386 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:39.171368 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5df9996869-cq6xm_24c16fd2-7c59-480f-a5b1-c52fffe7748d/manager/0.log" Sep 17 16:10:39.678332 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:39.678302 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7698c4fdd9-6ghjq_d85792b7-d7b1-40e7-97b1-9cd8002906d1/maas-api/0.log" Sep 17 16:10:41.002820 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:41.002791 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f85fc86df-hl4bd_3898c87d-f4a6-4ce0-8190-a08ed2ca28e9/authorino/0.log" Sep 17 16:10:41.123385 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:41.123357 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-2ttvj_54d0c38e-a5d1-4413-a14e-738e86f064ba/manager/0.log" Sep 17 16:10:41.243100 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:41.243071 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:10:41.372857 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:41.372832 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-kknfb_899f6280-8bbb-41ca-86a7-9cb7f95ada73/kuadrant-console-plugin/0.log" Sep 17 16:10:42.546741 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:42.546712 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dc8ddcd98-zgj6j_c70d3c6a-33cc-465c-996f-96e2608fee43/kube-auth-proxy/0.log" Sep 17 16:10:43.583131 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:43.583100 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-x8rtt_226f41ff-3630-4cb3-9635-64f6cc9cc16d/main/0.log" Sep 17 16:10:50.210927 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.210879 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-25lkm/must-gather-xtmrf"] Sep 17 16:10:50.211414 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.211203 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f81d981-d12a-4541-b0fd-ab2f1e99d6f7" containerName="maas-api" Sep 17 16:10:50.211414 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.211214 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f81d981-d12a-4541-b0fd-ab2f1e99d6f7" containerName="maas-api" Sep 17 16:10:50.211414 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.211276 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f81d981-d12a-4541-b0fd-ab2f1e99d6f7" containerName="maas-api" Sep 17 16:10:50.214485 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.214465 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25lkm/must-gather-xtmrf" Sep 17 16:10:50.218868 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.218818 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-25lkm\"/\"kube-root-ca.crt\"" Sep 17 16:10:50.218868 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.218840 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-25lkm\"/\"openshift-service-ca.crt\"" Sep 17 16:10:50.219358 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.219338 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-25lkm\"/\"default-dockercfg-bmf5g\"" Sep 17 16:10:50.230664 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.230639 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25lkm/must-gather-xtmrf"] Sep 17 16:10:50.244899 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.244877 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn8vl\" (UniqueName: \"kubernetes.io/projected/cd274c38-0296-42b6-a54e-d8cd081d88b1-kube-api-access-hn8vl\") pod \"must-gather-xtmrf\" (UID: \"cd274c38-0296-42b6-a54e-d8cd081d88b1\") " pod="openshift-must-gather-25lkm/must-gather-xtmrf" Sep 17 16:10:50.245008 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.244906 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd274c38-0296-42b6-a54e-d8cd081d88b1-must-gather-output\") pod \"must-gather-xtmrf\" (UID: \"cd274c38-0296-42b6-a54e-d8cd081d88b1\") " pod="openshift-must-gather-25lkm/must-gather-xtmrf" Sep 17 16:10:50.267373 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.267343 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:10:50.345506 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.345480 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn8vl\" (UniqueName: \"kubernetes.io/projected/cd274c38-0296-42b6-a54e-d8cd081d88b1-kube-api-access-hn8vl\") pod \"must-gather-xtmrf\" (UID: \"cd274c38-0296-42b6-a54e-d8cd081d88b1\") " pod="openshift-must-gather-25lkm/must-gather-xtmrf" Sep 17 16:10:50.345628 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.345511 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd274c38-0296-42b6-a54e-d8cd081d88b1-must-gather-output\") pod \"must-gather-xtmrf\" (UID: \"cd274c38-0296-42b6-a54e-d8cd081d88b1\") " pod="openshift-must-gather-25lkm/must-gather-xtmrf" Sep 17 16:10:50.346190 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.346139 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cd274c38-0296-42b6-a54e-d8cd081d88b1-must-gather-output\") pod \"must-gather-xtmrf\" (UID: \"cd274c38-0296-42b6-a54e-d8cd081d88b1\") " pod="openshift-must-gather-25lkm/must-gather-xtmrf" Sep 17 16:10:50.355402 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.355379 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn8vl\" (UniqueName: \"kubernetes.io/projected/cd274c38-0296-42b6-a54e-d8cd081d88b1-kube-api-access-hn8vl\") pod \"must-gather-xtmrf\" (UID: \"cd274c38-0296-42b6-a54e-d8cd081d88b1\") " pod="openshift-must-gather-25lkm/must-gather-xtmrf" Sep 17 16:10:50.359228 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.359207 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" limit=1024 Sep 17 16:10:50.523219 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.523135 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25lkm/must-gather-xtmrf" Sep 17 16:10:50.800933 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:50.800897 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25lkm/must-gather-xtmrf"] Sep 17 16:10:50.803598 ip-10-0-128-15 kubenswrapper[2581]: W0917 16:10:50.803568 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd274c38_0296_42b6_a54e_d8cd081d88b1.slice/crio-950003db249157013083575096b175ac88caeaddd663c42bda7e46a8e38157d4 WatchSource:0}: Error finding container 950003db249157013083575096b175ac88caeaddd663c42bda7e46a8e38157d4: Status 404 returned error can't find the container with id 950003db249157013083575096b175ac88caeaddd663c42bda7e46a8e38157d4 Sep 17 16:10:51.703089 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:51.703052 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25lkm/must-gather-xtmrf" event={"ID":"cd274c38-0296-42b6-a54e-d8cd081d88b1","Type":"ContainerStarted","Data":"950003db249157013083575096b175ac88caeaddd663c42bda7e46a8e38157d4"} Sep 17 16:10:52.710058 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:52.710025 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25lkm/must-gather-xtmrf" event={"ID":"cd274c38-0296-42b6-a54e-d8cd081d88b1","Type":"ContainerStarted","Data":"8f14f8e321cf69d2bf1bd67b6f617d7551b9ce72b363f8b29f67db7b288baf3b"} Sep 17 16:10:52.710058 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:52.710068 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25lkm/must-gather-xtmrf" event={"ID":"cd274c38-0296-42b6-a54e-d8cd081d88b1","Type":"ContainerStarted","Data":"dca329f1b8259599a73766e736e4d45a8347309c215d86b9cb4a6a904638dc7d"} Sep 17 16:10:52.735724 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:52.735659 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-25lkm/must-gather-xtmrf" podStartSLOduration=1.865502161 podStartE2EDuration="2.73563664s" totalImagesPullingTime="870.134479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:10:52.732304772 +0000 UTC m=+2439.728541518" watchObservedRunningTime="2026-09-17 16:10:52.73563664 +0000 UTC m=+2439.731873388" Sep 17 16:10:53.594608 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:53.594581 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5rf6m_721e805e-1d0a-473c-be1a-871d62cfebbe/global-pull-secret-syncer/0.log" Sep 17 16:10:53.773212 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:53.773176 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lv4s5_18b5b4bc-a399-468b-9957-2d51f1f24c6c/konnectivity-agent/0.log" Sep 17 16:10:53.910310 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:53.910221 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-15.ec2.internal_e96dc0286b5d333a4741dcc145860b91/haproxy/0.log" Sep 17 16:10:58.113521 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:58.113362 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f85fc86df-hl4bd_3898c87d-f4a6-4ce0-8190-a08ed2ca28e9/authorino/0.log" Sep 17 16:10:58.221743 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:58.221694 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-2ttvj_54d0c38e-a5d1-4413-a14e-738e86f064ba/manager/0.log" Sep 17 16:10:58.349686 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:10:58.349658 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-kknfb_899f6280-8bbb-41ca-86a7-9cb7f95ada73/kuadrant-console-plugin/0.log" Sep 17 16:11:00.562880 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:00.562845 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-przmb_66629cca-8fa3-4a89-97ec-dc73a66d9022/kube-state-metrics/0.log" Sep 17 16:11:00.594811 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:00.594729 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-przmb_66629cca-8fa3-4a89-97ec-dc73a66d9022/kube-rbac-proxy-main/0.log" Sep 17 16:11:00.638140 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:00.638110 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-przmb_66629cca-8fa3-4a89-97ec-dc73a66d9022/kube-rbac-proxy-self/0.log" Sep 17 16:11:00.743078 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:00.743050 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8rdxm_3982a855-709a-4b05-b4ee-a2d74565b64e/node-exporter/0.log" Sep 17 16:11:00.776136 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:00.776104 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8rdxm_3982a855-709a-4b05-b4ee-a2d74565b64e/kube-rbac-proxy/0.log" Sep 17 16:11:00.812298 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:00.812212 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8rdxm_3982a855-709a-4b05-b4ee-a2d74565b64e/init-textfile/0.log" Sep 17 16:11:01.067451 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:01.067342 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-q7rdh_60318aea-c20b-411a-86ec-3aa0b94d2edc/kube-rbac-proxy-main/0.log" Sep 17 16:11:01.103378 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:01.103276 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-q7rdh_60318aea-c20b-411a-86ec-3aa0b94d2edc/kube-rbac-proxy-self/0.log" Sep 17 16:11:01.136539 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:01.136466 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-q7rdh_60318aea-c20b-411a-86ec-3aa0b94d2edc/openshift-state-metrics/0.log" Sep 17 16:11:01.511298 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:01.511213 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-rkhk7_22787295-1701-4fc1-ab50-469300195ea0/prometheus-operator-admission-webhook/0.log" Sep 17 16:11:01.549704 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:01.549657 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74dd668c7f-gpk2h_36d804e1-cd53-437b-941c-4adfa0d28648/telemeter-client/0.log" Sep 17 16:11:01.574520 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:01.574473 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74dd668c7f-gpk2h_36d804e1-cd53-437b-941c-4adfa0d28648/reload/0.log" Sep 17 16:11:01.602813 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:01.602765 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74dd668c7f-gpk2h_36d804e1-cd53-437b-941c-4adfa0d28648/kube-rbac-proxy/0.log" Sep 17 16:11:03.599152 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.599118 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr"] Sep 17 16:11:03.606217 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.606190 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.618733 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.618705 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr"] Sep 17 16:11:03.685123 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.685091 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/234901a9-a501-420e-ba4e-27d78fd3d936-podres\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.685389 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.685370 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x2q5\" (UniqueName: \"kubernetes.io/projected/234901a9-a501-420e-ba4e-27d78fd3d936-kube-api-access-5x2q5\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.685582 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.685561 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/234901a9-a501-420e-ba4e-27d78fd3d936-proc\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.685775 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.685755 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/234901a9-a501-420e-ba4e-27d78fd3d936-sys\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.685936 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.685902 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/234901a9-a501-420e-ba4e-27d78fd3d936-lib-modules\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.787402 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.787358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/234901a9-a501-420e-ba4e-27d78fd3d936-podres\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.787594 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.787425 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x2q5\" (UniqueName: \"kubernetes.io/projected/234901a9-a501-420e-ba4e-27d78fd3d936-kube-api-access-5x2q5\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.787594 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.787477 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/234901a9-a501-420e-ba4e-27d78fd3d936-proc\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.787594 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.787521 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/234901a9-a501-420e-ba4e-27d78fd3d936-sys\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.787594 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.787557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/234901a9-a501-420e-ba4e-27d78fd3d936-lib-modules\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.787855 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.787763 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/234901a9-a501-420e-ba4e-27d78fd3d936-lib-modules\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.787855 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.787824 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/234901a9-a501-420e-ba4e-27d78fd3d936-proc\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.787986 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.787866 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/234901a9-a501-420e-ba4e-27d78fd3d936-sys\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.787986 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.787869 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/234901a9-a501-420e-ba4e-27d78fd3d936-podres\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.796442 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.796416 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x2q5\" (UniqueName: \"kubernetes.io/projected/234901a9-a501-420e-ba4e-27d78fd3d936-kube-api-access-5x2q5\") pod \"perf-node-gather-daemonset-75mzr\" (UID: \"234901a9-a501-420e-ba4e-27d78fd3d936\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:03.919822 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:03.919736 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:04.240569 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:04.240533 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr"] Sep 17 16:11:04.242970 ip-10-0-128-15 kubenswrapper[2581]: W0917 16:11:04.242940 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod234901a9_a501_420e_ba4e_27d78fd3d936.slice/crio-370bfc3104ee06d8f3d3da9aceeab181ee8ec411490f0732b4ec8d9466200279 WatchSource:0}: Error finding container 370bfc3104ee06d8f3d3da9aceeab181ee8ec411490f0732b4ec8d9466200279: Status 404 returned error can't find the container with id 370bfc3104ee06d8f3d3da9aceeab181ee8ec411490f0732b4ec8d9466200279 Sep 17 16:11:04.794102 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:04.793389 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" event={"ID":"234901a9-a501-420e-ba4e-27d78fd3d936","Type":"ContainerStarted","Data":"39d948779d5386e9932f9b21ead4d07132ff024b07e6576991c7d186c58cceaf"} Sep 17 16:11:04.794102 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:04.793462 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" event={"ID":"234901a9-a501-420e-ba4e-27d78fd3d936","Type":"ContainerStarted","Data":"370bfc3104ee06d8f3d3da9aceeab181ee8ec411490f0732b4ec8d9466200279"} Sep 17 16:11:04.794102 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:04.794017 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:04.816639 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:04.816576 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" podStartSLOduration=1.816553661 podStartE2EDuration="1.816553661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:11:04.815321275 +0000 UTC m=+2451.811558023" watchObservedRunningTime="2026-09-17 16:11:04.816553661 +0000 UTC m=+2451.812790420" Sep 17 16:11:06.605401 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:06.605374 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h5587_a766297c-8356-4831-815f-52186c63cd92/dns/0.log" Sep 17 16:11:06.636589 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:06.636555 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h5587_a766297c-8356-4831-815f-52186c63cd92/kube-rbac-proxy/0.log" Sep 17 16:11:06.871053 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:06.870985 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wd878_30c3098f-6b70-43ef-96fe-67a7c9f4ff91/dns-node-resolver/0.log" Sep 17 16:11:07.510286 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:07.510215 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65d59949f4-lb8rv_06863d44-90d7-4d15-a6df-5597dcb22945/registry/0.log" Sep 17 16:11:07.563670 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:07.563641 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d7hx7_0402d124-60fa-4a15-86cc-cac6e6904bb3/node-ca/0.log" Sep 17 16:11:09.236616 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:09.236581 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dc8ddcd98-zgj6j_c70d3c6a-33cc-465c-996f-96e2608fee43/kube-auth-proxy/0.log" Sep 17 16:11:10.744615 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:10.744575 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jcm4q_78acfb4b-12cd-4266-9f36-3c13f8a18263/serve-healthcheck-canary/0.log" Sep 17 16:11:10.816893 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:10.816860 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-75mzr" Sep 17 16:11:11.420126 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:11.420093 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ds2t5_b3816cff-380c-4f26-bc82-a648050bea57/kube-rbac-proxy/0.log" Sep 17 16:11:11.447839 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:11.447809 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ds2t5_b3816cff-380c-4f26-bc82-a648050bea57/exporter/0.log" Sep 17 16:11:11.474724 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:11.474698 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ds2t5_b3816cff-380c-4f26-bc82-a648050bea57/extractor/0.log" Sep 17 16:11:14.383268 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:14.383229 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dc8ddcd98-zgj6j_c70d3c6a-33cc-465c-996f-96e2608fee43/kube-auth-proxy/0.log" Sep 17 16:11:14.904183 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:14.904129 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5df9996869-cq6xm_24c16fd2-7c59-480f-a5b1-c52fffe7748d/manager/0.log" Sep 17 16:11:14.935907 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:14.935871 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5df9996869-cq6xm_24c16fd2-7c59-480f-a5b1-c52fffe7748d/copy-manifests/0.log" Sep 17 16:11:16.323756 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:16.323656 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c7d57fb89-wfn84_3ac1f6b4-4924-4b0f-9069-1bbabcfda87c/manager/0.log" Sep 17 16:11:16.489248 ip-10-0-128-15 kubenswrapper[2581]: I0917 16:11:16.489209 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dc8ddcd98-zgj6j_c70d3c6a-33cc-465c-996f-96e2608fee43/kube-auth-proxy/0.log"