Sep 11 15:06:33.785322 ip-10-0-134-128 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 15:06:33.785333 ip-10-0-134-128 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 15:06:33.785342 ip-10-0-134-128 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 15:06:33.785754 ip-10-0-134-128 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 15:06:43.536621 ip-10-0-134-128 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 15:06:43.536630 ip-10-0-134-128 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 37b13c9973584d38862beb1ce0a1671d -- Sep 11 15:08:25.641035 ip-10-0-134-128 systemd[1]: Starting Kubernetes Kubelet... Sep 11 15:08:26.091136 ip-10-0-134-128 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 15:08:26.091136 ip-10-0-134-128 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 15:08:26.091136 ip-10-0-134-128 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 15:08:26.091136 ip-10-0-134-128 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 15:08:26.091136 ip-10-0-134-128 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 15:08:26.093210 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.093125 2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 15:08:26.098432 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098409 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:08:26.098432 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098429 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:08:26.098432 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098433 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:08:26.098432 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098437 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:08:26.098432 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098440 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098444 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098449 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098453 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098456 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098459 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098462 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098465 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098468 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098471 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098473 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098476 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098479 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098482 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098484 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098487 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098490 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098492 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098495 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098497 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:08:26.098631 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098500 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098503 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098505 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098508 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098510 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098513 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098515 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098523 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098525 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098528 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098531 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098533 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098536 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098539 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098542 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098545 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098548 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098550 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098554 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:08:26.099129 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098557 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098560 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098562 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098566 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098571 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098574 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098578 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098580 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098583 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098586 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098588 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098591 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098594 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098597 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098599 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098602 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098605 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098608 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098610 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098613 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:08:26.099600 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098616 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098619 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098621 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098624 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098627 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098630 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098634 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098638 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098642 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098645 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098648 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098651 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098653 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098656 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098659 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098662 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098664 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098667 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098669 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:08:26.100093 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098672 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098675 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098678 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098680 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098683 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098686 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098688 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098691 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098693 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098696 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098699 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098701 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098704 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098708 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098711 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098713 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.098716 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099439 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099446 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:08:26.100554 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099449 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099454 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099458 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099461 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099464 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099467 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099470 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099473 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099476 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099479 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099481 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099484 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099487 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099489 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099492 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099499 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099501 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099504 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099508 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:08:26.101071 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099511 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099514 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099517 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099520 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099522 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099525 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099527 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099531 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099534 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099536 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099538 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099541 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099544 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099547 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099549 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099552 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099554 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099557 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099560 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099562 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:08:26.101548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099565 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099567 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099569 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099572 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099575 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099578 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099580 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099583 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099585 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099588 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099591 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099594 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099596 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099598 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099601 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099605 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099609 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099612 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099616 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:08:26.102045 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099619 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099622 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099625 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099628 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099630 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099633 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099636 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099638 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099641 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099644 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099646 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099649 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099652 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099655 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099657 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099660 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099662 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099665 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099667 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099669 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:08:26.102518 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099672 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099675 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099677 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099680 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099683 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099685 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099688 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099690 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099693 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099696 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099699 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099701 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099704 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099707 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099710 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099712 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099715 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099717 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.099720 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100096 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:08:26.103019 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100101 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100105 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100108 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100110 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100113 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100116 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100119 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100121 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100124 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100127 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100129 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100132 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100134 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100137 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100139 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100142 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100145 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100147 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100150 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100153 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:08:26.103512 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100155 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100158 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100161 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100163 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100167 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100169 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100172 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100174 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100177 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100179 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100182 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100184 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100187 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100190 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100192 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100196 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100198 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100201 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100203 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100205 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:08:26.104018 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100208 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100210 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100213 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100215 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100218 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100220 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100223 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100225 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100228 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100231 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100233 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100236 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100238 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100241 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100244 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100246 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100248 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100251 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100254 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:08:26.104505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100258 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100262 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100264 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100267 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100270 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100273 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100276 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100279 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100282 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100284 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100287 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100290 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100292 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100295 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100299 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100302 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100306 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100308 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100313 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:08:26.104981 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100316 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100319 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100322 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100325 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100328 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100330 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100333 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100335 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100338 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100340 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100343 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100345 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100348 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100350 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100353 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100355 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100358 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100360 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100362 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.100365 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:08:26.105441 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100443 2558 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100486 2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100493 2558 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100497 2558 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100506 2558 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100509 2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100514 2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100518 2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100521 2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100524 2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100528 2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100531 2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100534 2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100537 2558 flags.go:64] FLAG: --cgroup-root="" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100540 2558 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100543 2558 flags.go:64] FLAG: --client-ca-file="" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100547 2558 flags.go:64] FLAG: --cloud-provider="external" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100551 2558 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100555 2558 flags.go:64] FLAG: --cluster-domain="" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100558 2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100561 2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100565 2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100568 2558 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100573 2558 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 15:08:26.105974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100576 2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100579 2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100582 2558 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100585 2558 flags.go:64] FLAG: --contention-profiling="false" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100588 2558 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100591 2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100594 2558 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100597 2558 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100601 2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100604 2558 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100607 2558 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100610 2558 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100613 2558 flags.go:64] FLAG: --enable-server="true" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100616 2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100621 2558 flags.go:64] FLAG: --event-burst="100" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100624 2558 flags.go:64] FLAG: --event-qps="50" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100627 2558 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100630 2558 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100634 2558 flags.go:64] FLAG: --eviction-hard="" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100637 2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100640 2558 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100643 2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100661 2558 flags.go:64] FLAG: --eviction-soft="" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100664 2558 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100670 2558 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 15:08:26.106552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100674 2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100677 2558 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100680 2558 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100683 2558 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100686 2558 flags.go:64] FLAG: --feature-gates="" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100690 2558 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100693 2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100696 2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100699 2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100702 2558 flags.go:64] FLAG: --healthz-port="10248" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100705 2558 flags.go:64] FLAG: --help="false" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100708 2558 flags.go:64] FLAG: --hostname-override="ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100711 2558 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100714 2558 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100717 2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100720 2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100724 2558 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100727 2558 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100730 2558 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100734 2558 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100749 2558 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100753 2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100757 2558 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100760 2558 flags.go:64] FLAG: --kube-reserved="" Sep 11 15:08:26.107176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100762 2558 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100765 2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100768 2558 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100771 2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100774 2558 flags.go:64] FLAG: --lock-file="" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100777 2558 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100780 2558 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100784 2558 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100790 2558 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100793 2558 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100796 2558 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100799 2558 flags.go:64] FLAG: --logging-format="text" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100802 2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100805 2558 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100808 2558 flags.go:64] FLAG: --manifest-url="" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100811 2558 flags.go:64] FLAG: --manifest-url-header="" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100816 2558 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100819 2558 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100822 2558 flags.go:64] FLAG: --max-pods="110" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100826 2558 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100829 2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100831 2558 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100835 2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100838 2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100841 2558 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 15:08:26.107784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100844 2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100852 2558 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100856 2558 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100859 2558 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100862 2558 flags.go:64] FLAG: --pod-cidr="" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100865 2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100870 2558 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100873 2558 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100876 2558 flags.go:64] FLAG: --pods-per-core="0" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100879 2558 flags.go:64] FLAG: --port="10250" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100882 2558 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100885 2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01145f2cac7fe085e" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100888 2558 flags.go:64] FLAG: --qos-reserved="" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100891 2558 flags.go:64] FLAG: --read-only-port="10255" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100894 2558 flags.go:64] FLAG: --register-node="true" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100898 2558 flags.go:64] FLAG: --register-with-taints="" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100902 2558 flags.go:64] FLAG: --registry-burst="10" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100905 2558 flags.go:64] FLAG: --registry-qps="5" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100908 2558 flags.go:64] FLAG: --reserved-cpus="" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100911 2558 flags.go:64] FLAG: --reserved-memory="" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100915 2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100918 2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100921 2558 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100924 2558 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100927 2558 flags.go:64] FLAG: --runonce="false" Sep 11 15:08:26.108389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100930 2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100933 2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100936 2558 flags.go:64] FLAG: --seccomp-default="false" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100939 2558 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100941 2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100945 2558 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100948 2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100951 2558 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100953 2558 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100958 2558 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100961 2558 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100964 2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100967 2558 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100970 2558 flags.go:64] FLAG: --system-cgroups="" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100973 2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100978 2558 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100981 2558 flags.go:64] FLAG: --tls-cert-file="" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100984 2558 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100988 2558 flags.go:64] FLAG: --tls-min-version="" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100991 2558 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.100994 2558 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.101004 2558 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.101011 2558 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.101014 2558 flags.go:64] FLAG: --v="2" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.101018 2558 flags.go:64] FLAG: --version="false" Sep 11 15:08:26.109073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.101023 2558 flags.go:64] FLAG: --vmodule="" Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.101026 2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.101030 2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101146 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101150 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101153 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101156 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101159 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101162 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101164 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101167 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101170 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101172 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101175 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101177 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101180 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101184 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101187 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101190 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101193 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:08:26.109711 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101195 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101198 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101201 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101205 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101209 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101212 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101215 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101217 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101220 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101224 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101227 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101230 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101232 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101237 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101240 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101243 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101246 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101249 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101252 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:08:26.110224 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101254 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101257 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101259 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101262 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101264 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101267 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101269 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101272 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101275 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101279 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101281 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101284 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101287 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101289 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101292 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101294 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101297 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101299 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101302 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101311 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:08:26.110699 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101314 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101317 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101321 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101323 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101326 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101329 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101332 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101335 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101337 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101340 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101342 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101345 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101348 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101350 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101353 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101356 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101358 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101361 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101363 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101366 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:08:26.111276 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101368 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101372 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101375 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101387 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101391 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101394 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101396 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101399 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101402 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101405 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101407 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101410 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101414 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101418 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101422 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101425 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101428 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101430 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101433 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101435 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:08:26.111783 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101439 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101441 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.101444 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.102168 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.109979 2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.109994 2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.111057 2558 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.111065 2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111121 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111126 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111129 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111133 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111136 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111139 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111141 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:08:26.112271 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111145 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111147 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111150 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111153 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111156 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111159 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111161 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111164 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111167 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111170 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111173 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111177 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111181 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111185 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111189 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111192 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111195 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111198 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111201 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111204 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:08:26.112651 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111207 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111209 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111213 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111216 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111219 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111221 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111224 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111227 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111229 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111232 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111234 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111237 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111239 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111242 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111245 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111247 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111250 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111252 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111255 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111257 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:08:26.113216 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111260 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111263 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111266 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111269 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111271 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111274 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111277 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111280 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111283 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111286 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111289 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111292 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111295 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111297 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111300 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111303 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111306 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111308 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111311 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111313 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:08:26.113704 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111316 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111318 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111321 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111323 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111326 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111330 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111333 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111336 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111338 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111341 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111343 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111346 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111349 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111352 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111354 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111357 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111359 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111362 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111365 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:08:26.114204 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111367 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111369 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111372 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111375 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111378 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111380 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111383 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111386 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111388 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111391 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111393 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111396 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111399 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.111404 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111516 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:08:26.114722 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111521 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111525 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111529 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111532 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111535 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111538 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111541 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111544 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111547 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111550 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111553 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111556 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111560 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111562 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111565 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111568 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111570 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111573 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111576 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:08:26.115121 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111580 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111583 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111586 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111589 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111592 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111595 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111597 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111600 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111603 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111606 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111609 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111611 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111614 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111616 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111619 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111621 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111624 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111626 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111629 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111632 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:08:26.115612 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111634 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111637 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111639 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111643 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111645 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111648 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111651 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111653 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111656 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111658 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111662 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111665 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111668 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111670 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111673 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111675 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111678 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111681 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111683 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:08:26.116192 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111685 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111688 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111691 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111694 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111696 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111699 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111701 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111704 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111706 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111709 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111711 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111714 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111717 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111719 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111722 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111724 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111727 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111730 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111733 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111749 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:08:26.116664 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111753 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111756 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111759 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111761 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111764 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111766 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111769 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111772 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111774 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111777 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111779 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111782 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111785 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111787 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111790 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111793 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111796 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111799 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111801 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:08:26.117261 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:26.111804 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:08:26.117767 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.111809 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 15:08:26.117767 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.111912 2558 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 15:08:26.117767 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.114369 2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 15:08:26.117767 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.115521 2558 server.go:1019] "Starting client certificate rotation" Sep 11 15:08:26.117767 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.115634 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 15:08:26.117767 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.115676 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 15:08:26.118334 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.118320 2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 15:08:26.119563 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.119527 2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 15:08:26.134137 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.134114 2558 log.go:25] "Validated CRI v1 runtime API" Sep 11 15:08:26.137500 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.137483 2558 log.go:25] "Validated CRI v1 image API" Sep 11 15:08:26.138401 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.138385 2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 15:08:26.143447 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.143428 2558 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9fead45f-6b57-4a73-a768-c3817f81842c:/dev/nvme0n1p3 b2d4a430-20a2-4465-baf1-57eced6a2d94:/dev/nvme0n1p4] Sep 11 15:08:26.143501 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.143447 2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 15:08:26.148908 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.148800 2558 manager.go:216] Machine: {Timestamp:2026-09-11 15:08:26.147384561 +0000 UTC m=+0.379308983 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100861 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23efb439cc71574ee6cc31129e50a5 SystemUUID:ec23efb4-39cc-7157-4ee6-cc31129e50a5 BootID:37b13c99-7358-4d38-862b-eb1ce0a1671d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c4:d0:cb:fc:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c4:d0:cb:fc:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:6b:da:d1:8a:f7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 15:08:26.148908 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.148904 2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 15:08:26.149038 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.149026 2558 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 15:08:26.152602 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.152577 2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 15:08:26.152763 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.152604 2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-128.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 15:08:26.153193 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.153176 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 15:08:26.153471 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.153459 2558 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 15:08:26.153508 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.153474 2558 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 15:08:26.153508 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.153482 2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 15:08:26.154387 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.154377 2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 15:08:26.155636 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.155626 2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 15:08:26.160222 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.160211 2558 state_mem.go:36] "Initialized new in-memory state store" Sep 11 15:08:26.160330 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.160320 2558 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 15:08:26.160358 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.160351 2558 kubelet.go:486] "Attempting to sync node with API server" Sep 11 15:08:26.160413 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.160364 2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 15:08:26.160413 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.160386 2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 15:08:26.160413 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.160400 2558 kubelet.go:398] "Adding apiserver pod source" Sep 11 15:08:26.160413 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.160412 2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 15:08:26.160854 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.160841 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 15:08:26.160899 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.160860 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 15:08:26.163994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.163972 2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 15:08:26.166839 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.166825 2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 15:08:26.166916 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.166857 2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 15:08:26.170877 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.170861 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 15:08:26.170877 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.170880 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 15:08:26.170989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.170886 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 15:08:26.170989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.170892 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 15:08:26.170989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.170900 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 15:08:26.170989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.170906 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 15:08:26.170989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.170918 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 15:08:26.170989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.170924 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 15:08:26.170989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.170930 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 15:08:26.170989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.170935 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 15:08:26.170989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.170947 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 15:08:26.170989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.170953 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 15:08:26.171476 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.171444 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 15:08:26.171646 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.171625 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 15:08:26.172037 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.172027 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 15:08:26.172075 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.172037 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 15:08:26.175642 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.175628 2558 server.go:1268] "Started kubelet" Sep 11 15:08:26.175718 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.175694 2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 15:08:26.176001 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.175806 2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 15:08:26.176186 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.176169 2558 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 15:08:26.176560 ip-10-0-134-128 systemd[1]: Started Kubernetes Kubelet. Sep 11 15:08:26.177821 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.177785 2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 15:08:26.181634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.181590 2558 server.go:310] "Adding debug handlers to kubelet server" Sep 11 15:08:26.183137 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.183111 2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 15:08:26.186479 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.186461 2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-128.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 15:08:26.187672 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.186584 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-128.ec2.internal.18d44cb83e48d707 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-128.ec2.internal,UID:ip-10-0-134-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-128.ec2.internal,},FirstTimestamp:2026-09-11 15:08:26.175584007 +0000 UTC m=+0.407508429,LastTimestamp:2026-09-11 15:08:26.175584007 +0000 UTC m=+0.407508429,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-128.ec2.internal,}" Sep 11 15:08:26.188244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.188225 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 15:08:26.188979 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.188962 2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 15:08:26.189074 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.189041 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Sep 11 15:08:26.189710 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.189694 2558 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 15:08:26.189710 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.189711 2558 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 15:08:26.189864 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.189694 2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 15:08:26.189864 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.189824 2558 reconstruct.go:97] "Volume reconstruction finished" Sep 11 15:08:26.189864 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.189832 2558 reconciler.go:29] "Reconciler: start to sync state" Sep 11 15:08:26.190024 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.190011 2558 factory.go:153] Registering CRI-O factory Sep 11 15:08:26.190076 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.190043 2558 factory.go:223] Registration of the crio container factory successfully Sep 11 15:08:26.190155 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.190145 2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 15:08:26.190186 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.190157 2558 factory.go:55] Registering systemd factory Sep 11 15:08:26.190186 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.190166 2558 factory.go:223] Registration of the systemd container factory successfully Sep 11 15:08:26.190186 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.190181 2558 factory.go:103] Registering Raw factory Sep 11 15:08:26.190283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.190194 2558 manager.go:1196] Started watching for new ooms in manager Sep 11 15:08:26.190558 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.190543 2558 manager.go:349] Starting recovery of all containers Sep 11 15:08:26.192827 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.192807 2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 15:08:26.192938 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.192919 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 15:08:26.198688 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.198654 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 15:08:26.202387 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.202374 2558 manager.go:354] Recovery completed Sep 11 15:08:26.205446 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.205421 2558 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 11 15:08:26.208592 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.208578 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:26.210405 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.210389 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:26.210475 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.210424 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:26.210475 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.210437 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:26.210986 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.210967 2558 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 15:08:26.210986 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.210980 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:26.211166 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.210983 2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 15:08:26.211166 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.211061 2558 state_mem.go:36] "Initialized new in-memory state store" Sep 11 15:08:26.211293 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.211277 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7svgb" Sep 11 15:08:26.211823 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.211803 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:26.211922 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.211835 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:26.211922 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.211853 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:26.212635 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.212553 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-128.ec2.internal.18d44cb8405c2809 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-128.ec2.internal,UID:ip-10-0-134-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-128.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-128.ec2.internal,},FirstTimestamp:2026-09-11 15:08:26.210404361 +0000 UTC m=+0.442328783,LastTimestamp:2026-09-11 15:08:26.210404361 +0000 UTC m=+0.442328783,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-128.ec2.internal,}" Sep 11 15:08:26.213326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.213301 2558 policy_none.go:49] "None policy: Start" Sep 11 15:08:26.213326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.213326 2558 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 15:08:26.213429 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.213341 2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 15:08:26.215390 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.215377 2558 policy_none.go:47] "Start" Sep 11 15:08:26.218034 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.218018 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7svgb" Sep 11 15:08:26.225507 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.225442 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-128.ec2.internal.18d44cb8405c91f4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-128.ec2.internal,UID:ip-10-0-134-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-128.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-128.ec2.internal,},FirstTimestamp:2026-09-11 15:08:26.210431476 +0000 UTC m=+0.442355898,LastTimestamp:2026-09-11 15:08:26.210431476 +0000 UTC m=+0.442355898,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-128.ec2.internal,}" Sep 11 15:08:26.257392 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.257368 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.257413 2558 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.257433 2558 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.257468 2558 manager.go:337] "Starting Device Plugin manager" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.257471 2558 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.257545 2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.257561 2558 server.go:87] "Starting device plugin registration server" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.257977 2558 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.258003 2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.258105 2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.258190 2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.258200 2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.258749 2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.258799 2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-128.ec2.internal\" not found" Sep 11 15:08:26.269514 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.259497 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:08:26.358056 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.357949 2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal"] Sep 11 15:08:26.358056 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.358040 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:26.358229 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.358102 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:26.359066 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.359047 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:26.359185 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.359082 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:26.359185 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.359097 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:26.359185 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.359048 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:26.359185 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.359151 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:26.359185 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.359163 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:26.359185 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.359183 2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.361588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.361573 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:26.361753 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.361725 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.361809 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.361772 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:26.362333 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.362310 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:26.362424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.362338 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:26.362424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.362356 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:26.362424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.362364 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:26.362424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.362369 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:26.362424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.362378 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:26.364692 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.364673 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.364794 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.364704 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:26.365406 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.365379 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:26.365406 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.365403 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:26.365529 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.365417 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:26.368436 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.368419 2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.368505 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.368443 2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-128.ec2.internal\": node \"ip-10-0-134-128.ec2.internal\" not found" Sep 11 15:08:26.386371 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.386351 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-128.ec2.internal\" not found" node="ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.390660 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.390643 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-128.ec2.internal\" not found" node="ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.391682 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.391667 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e1779d60c22f5ff6cb06e66ddefda83-config\") pod \"kube-apiserver-proxy-ip-10-0-134-128.ec2.internal\" (UID: \"4e1779d60c22f5ff6cb06e66ddefda83\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.391734 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.391689 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ffa511d6fab2cc6c0e8967b38c06790-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal\" (UID: \"2ffa511d6fab2cc6c0e8967b38c06790\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.391734 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.391707 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ffa511d6fab2cc6c0e8967b38c06790-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal\" (UID: \"2ffa511d6fab2cc6c0e8967b38c06790\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.399015 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.398996 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Sep 11 15:08:26.492125 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.492094 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e1779d60c22f5ff6cb06e66ddefda83-config\") pod \"kube-apiserver-proxy-ip-10-0-134-128.ec2.internal\" (UID: \"4e1779d60c22f5ff6cb06e66ddefda83\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.492125 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.492124 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ffa511d6fab2cc6c0e8967b38c06790-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal\" (UID: \"2ffa511d6fab2cc6c0e8967b38c06790\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.492284 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.492142 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ffa511d6fab2cc6c0e8967b38c06790-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal\" (UID: \"2ffa511d6fab2cc6c0e8967b38c06790\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.492284 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.492194 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ffa511d6fab2cc6c0e8967b38c06790-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal\" (UID: \"2ffa511d6fab2cc6c0e8967b38c06790\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.492284 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.492212 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ffa511d6fab2cc6c0e8967b38c06790-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal\" (UID: \"2ffa511d6fab2cc6c0e8967b38c06790\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.492284 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.492207 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e1779d60c22f5ff6cb06e66ddefda83-config\") pod \"kube-apiserver-proxy-ip-10-0-134-128.ec2.internal\" (UID: \"4e1779d60c22f5ff6cb06e66ddefda83\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.499208 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.499188 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Sep 11 15:08:26.599814 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.599787 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Sep 11 15:08:26.690428 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.690366 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.694436 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:26.694412 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Sep 11 15:08:26.700622 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.700606 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Sep 11 15:08:26.801138 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.801103 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Sep 11 15:08:26.901791 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:26.901606 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Sep 11 15:08:27.002963 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:27.002911 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Sep 11 15:08:27.103519 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:27.103483 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Sep 11 15:08:27.114720 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.114689 2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 15:08:27.114920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.114897 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 15:08:27.204353 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:27.204321 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Sep 11 15:08:27.224374 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.224339 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 15:03:27 +0000 UTC" deadline="2028-05-07 10:44:45.283786965 +0000 UTC" Sep 11 15:08:27.224374 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.224370 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14491h36m18.059420484s" Sep 11 15:08:27.301260 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.301237 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:08:27.302435 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.302415 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:08:27.302557 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.302454 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:08:27.302557 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.302468 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:08:27.304700 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:27.304680 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-128.ec2.internal\" not found" Sep 11 15:08:27.309911 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.309893 2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:08:27.389219 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.389192 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" Sep 11 15:08:27.404854 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:27.404817 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffa511d6fab2cc6c0e8967b38c06790.slice/crio-56178fc54067f401572ea09fee261eb7dfae7eda9af20393a489699b9ad53f74 WatchSource:0}: Error finding container 56178fc54067f401572ea09fee261eb7dfae7eda9af20393a489699b9ad53f74: Status 404 returned error can't find the container with id 56178fc54067f401572ea09fee261eb7dfae7eda9af20393a489699b9ad53f74 Sep 11 15:08:27.408102 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.407960 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 15:08:27.410264 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.410243 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" Sep 11 15:08:27.410406 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.410387 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 15:08:27.411596 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:27.411574 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1779d60c22f5ff6cb06e66ddefda83.slice/crio-58f7d8e6e45d4fd26a303ba79d5957e3caaeac961b60b21515cc7f4788edac20 WatchSource:0}: Error finding container 58f7d8e6e45d4fd26a303ba79d5957e3caaeac961b60b21515cc7f4788edac20: Status 404 returned error can't find the container with id 58f7d8e6e45d4fd26a303ba79d5957e3caaeac961b60b21515cc7f4788edac20 Sep 11 15:08:27.421041 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.421025 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 15:08:27.424472 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.424447 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:08:27.766283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:27.766200 2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:08:28.163780 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.163688 2558 apiserver.go:52] "Watching apiserver" Sep 11 15:08:28.174604 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.174571 2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 15:08:28.175906 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.175873 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-5hxt6","kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj","openshift-cluster-node-tuning-operator/tuned-pzmbs","openshift-dns/node-resolver-r85xv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal","openshift-multus/multus-additional-cni-plugins-6b84v","openshift-multus/multus-skzwk","kube-system/global-pull-secret-syncer-cdnlq","openshift-image-registry/node-ca-4mzss","openshift-multus/network-metrics-daemon-lkx5k","openshift-network-diagnostics/network-check-target-qcg9p","openshift-network-operator/iptables-alerter-dm7vf","openshift-ovn-kubernetes/ovnkube-node-jbxnk"] Sep 11 15:08:28.178623 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.178601 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.181025 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.181002 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dm7vf" Sep 11 15:08:28.181597 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.181571 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 15:08:28.181694 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.181574 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 15:08:28.181753 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.181720 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 15:08:28.181832 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.181738 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 15:08:28.181968 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.181947 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pjgqz\"" Sep 11 15:08:28.182329 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.182314 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 15:08:28.183355 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.183331 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m78gj\"" Sep 11 15:08:28.184318 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.184290 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:08:28.184413 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.184369 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 15:08:28.184636 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.184617 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 15:08:28.184825 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.184803 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.187222 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.187201 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 15:08:28.187586 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.187432 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 15:08:28.187765 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.187663 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 15:08:28.187989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.187897 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2xlmh\"" Sep 11 15:08:28.188100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.188048 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.189097 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.189076 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 15:08:28.189842 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.189822 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r85xv" Sep 11 15:08:28.190396 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.190375 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5mjvx\"" Sep 11 15:08:28.190496 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.190402 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 15:08:28.190549 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.190536 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:08:28.191996 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.191975 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 15:08:28.192381 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.192359 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rj2ft\"" Sep 11 15:08:28.192473 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.192438 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.192473 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.192466 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:08:28.193099 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.193070 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 15:08:28.194645 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.194624 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 15:08:28.194897 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.194879 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r6bcz\"" Sep 11 15:08:28.195015 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.194934 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 15:08:28.195201 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.195183 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:28.195335 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.195298 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:28.195661 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.195638 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 15:08:28.195733 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.195684 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 15:08:28.196267 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.196247 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 15:08:28.196890 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.196869 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 15:08:28.198470 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.198454 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5hxt6" Sep 11 15:08:28.201003 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.200973 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 15:08:28.201230 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.201213 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qhkst\"" Sep 11 15:08:28.201331 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.201314 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 15:08:28.201539 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.201478 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.201998 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.201858 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmpdp\" (UniqueName: \"kubernetes.io/projected/d209e966-9242-4f9d-8d2d-63dc61ffc5c3-kube-api-access-cmpdp\") pod \"node-resolver-r85xv\" (UID: \"d209e966-9242-4f9d-8d2d-63dc61ffc5c3\") " pod="openshift-dns/node-resolver-r85xv" Sep 11 15:08:28.201998 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.201945 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-slash\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.201998 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.201977 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac2f5397-4638-4694-9857-e5a25f186318-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.201998 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202001 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-registration-dir\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.202244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202016 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-var-lib-kubelet\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.202244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202031 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-run-netns\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.202244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202083 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d209e966-9242-4f9d-8d2d-63dc61ffc5c3-hosts-file\") pod \"node-resolver-r85xv\" (UID: \"d209e966-9242-4f9d-8d2d-63dc61ffc5c3\") " pod="openshift-dns/node-resolver-r85xv" Sep 11 15:08:28.202244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202156 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-node-log\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.202244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202180 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-sys\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.202483 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202205 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-run-openvswitch\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.202483 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202281 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e77adf8c-7131-49f1-b248-11a48c132170-env-overrides\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.202483 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202307 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szsf\" (UniqueName: \"kubernetes.io/projected/e77adf8c-7131-49f1-b248-11a48c132170-kube-api-access-4szsf\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.202483 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202361 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-kubelet\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.202483 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202379 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e77adf8c-7131-49f1-b248-11a48c132170-ovnkube-config\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.202483 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202424 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e77adf8c-7131-49f1-b248-11a48c132170-ovn-node-metrics-cert\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.202483 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202452 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac2f5397-4638-4694-9857-e5a25f186318-cnibin\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.202483 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202477 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-sys-fs\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.202914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202497 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e182034-0cd8-47b8-9c49-c872d2053550-host-slash\") pod \"iptables-alerter-dm7vf\" (UID: \"6e182034-0cd8-47b8-9c49-c872d2053550\") " pod="openshift-network-operator/iptables-alerter-dm7vf" Sep 11 15:08:28.202914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202520 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-etc-selinux\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.202914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202544 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-lib-modules\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.202914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202594 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-run-systemd\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.202914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202622 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn6x8\" (UniqueName: \"kubernetes.io/projected/ac2f5397-4638-4694-9857-e5a25f186318-kube-api-access-jn6x8\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.202914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202652 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-device-dir\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.202914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202683 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-host\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.202914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202755 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d209e966-9242-4f9d-8d2d-63dc61ffc5c3-tmp-dir\") pod \"node-resolver-r85xv\" (UID: \"d209e966-9242-4f9d-8d2d-63dc61ffc5c3\") " pod="openshift-dns/node-resolver-r85xv" Sep 11 15:08:28.202914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202811 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.202914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202855 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.202914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202891 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac2f5397-4638-4694-9857-e5a25f186318-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.202914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202913 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-sysctl-d\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202954 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-cni-bin\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.202995 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-sysctl-conf\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203026 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-sysconfig\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203052 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-log-socket\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203079 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac2f5397-4638-4694-9857-e5a25f186318-system-cni-dir\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203125 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vzn9\" (UniqueName: \"kubernetes.io/projected/6e182034-0cd8-47b8-9c49-c872d2053550-kube-api-access-4vzn9\") pod \"iptables-alerter-dm7vf\" (UID: \"6e182034-0cd8-47b8-9c49-c872d2053550\") " pod="openshift-network-operator/iptables-alerter-dm7vf" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203205 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203215 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-systemd\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203259 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-var-lib-openvswitch\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203292 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79384694-7eec-4e99-a246-0799138642be-tmp\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203334 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-etc-openvswitch\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203358 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-run-ovn\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203422 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-cni-netd\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203449 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e77adf8c-7131-49f1-b248-11a48c132170-ovnkube-script-lib\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.203507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203496 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-run\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.204271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203533 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.204271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203556 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac2f5397-4638-4694-9857-e5a25f186318-cni-binary-copy\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.204271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203581 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac2f5397-4638-4694-9857-e5a25f186318-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.204271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203629 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-modprobe-d\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.204271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203655 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-systemd-units\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.204271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203679 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e182034-0cd8-47b8-9c49-c872d2053550-iptables-alerter-script\") pod \"iptables-alerter-dm7vf\" (UID: \"6e182034-0cd8-47b8-9c49-c872d2053550\") " pod="openshift-network-operator/iptables-alerter-dm7vf" Sep 11 15:08:28.204271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203709 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9f7s\" (UniqueName: \"kubernetes.io/projected/1c7b0712-3366-4635-b6b8-386c025c3da3-kube-api-access-r9f7s\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.204271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203732 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k76rk\" (UniqueName: \"kubernetes.io/projected/79384694-7eec-4e99-a246-0799138642be-kube-api-access-k76rk\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.204271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203758 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-kubernetes\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.204271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203804 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79384694-7eec-4e99-a246-0799138642be-etc-tuned\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.204271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.203862 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-socket-dir\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.204271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.204079 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4mzss" Sep 11 15:08:28.208643 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.207898 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v64wk\"" Sep 11 15:08:28.208643 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.208140 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 15:08:28.208643 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.208469 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xpmng\"" Sep 11 15:08:28.208928 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.208893 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:28.209039 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.208987 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:28.210349 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.210139 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 15:08:28.210349 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.210171 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 15:08:28.210349 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.209955 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 15:08:28.212452 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.212092 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:28.212452 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.212156 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:28.230316 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.230293 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8l7wx" Sep 11 15:08:28.237935 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.237912 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8l7wx" Sep 11 15:08:28.264258 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.264202 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" event={"ID":"4e1779d60c22f5ff6cb06e66ddefda83","Type":"ContainerStarted","Data":"58f7d8e6e45d4fd26a303ba79d5957e3caaeac961b60b21515cc7f4788edac20"} Sep 11 15:08:28.265688 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.265661 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" event={"ID":"2ffa511d6fab2cc6c0e8967b38c06790","Type":"ContainerStarted","Data":"56178fc54067f401572ea09fee261eb7dfae7eda9af20393a489699b9ad53f74"} Sep 11 15:08:28.291205 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.291183 2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 15:08:28.304678 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.304647 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-modprobe-d\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.304824 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.304691 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-systemd-units\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.304824 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.304722 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-run-multus-certs\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.304824 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.304751 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dz6w\" (UniqueName: \"kubernetes.io/projected/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-kube-api-access-9dz6w\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:28.304824 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.304796 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e182034-0cd8-47b8-9c49-c872d2053550-iptables-alerter-script\") pod \"iptables-alerter-dm7vf\" (UID: \"6e182034-0cd8-47b8-9c49-c872d2053550\") " pod="openshift-network-operator/iptables-alerter-dm7vf" Sep 11 15:08:28.304824 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.304801 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-systemd-units\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.304824 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.304820 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9f7s\" (UniqueName: \"kubernetes.io/projected/1c7b0712-3366-4635-b6b8-386c025c3da3-kube-api-access-r9f7s\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.305142 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.304846 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k76rk\" (UniqueName: \"kubernetes.io/projected/79384694-7eec-4e99-a246-0799138642be-kube-api-access-k76rk\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.305142 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.304863 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-modprobe-d\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.305142 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.304919 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f547eaf0-def0-4524-9755-e682cd106e33-kubelet-config\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:28.305142 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.304946 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb-serviceca\") pod \"node-ca-4mzss\" (UID: \"6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb\") " pod="openshift-image-registry/node-ca-4mzss" Sep 11 15:08:28.305142 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305007 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-kubernetes\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.305142 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305052 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79384694-7eec-4e99-a246-0799138642be-etc-tuned\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.305142 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305091 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-etc-kubernetes\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.305142 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305122 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-socket-dir\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.305142 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305128 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-kubernetes\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305170 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmpdp\" (UniqueName: \"kubernetes.io/projected/d209e966-9242-4f9d-8d2d-63dc61ffc5c3-kube-api-access-cmpdp\") pod \"node-resolver-r85xv\" (UID: \"d209e966-9242-4f9d-8d2d-63dc61ffc5c3\") " pod="openshift-dns/node-resolver-r85xv" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305217 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-slash\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305243 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-run-k8s-cni-cncf-io\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305262 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-socket-dir\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305299 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-slash\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305361 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac2f5397-4638-4694-9857-e5a25f186318-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305395 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-registration-dir\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305411 2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305425 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-var-lib-kubelet\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305454 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-run-netns\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305469 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e182034-0cd8-47b8-9c49-c872d2053550-iptables-alerter-script\") pod \"iptables-alerter-dm7vf\" (UID: \"6e182034-0cd8-47b8-9c49-c872d2053550\") " pod="openshift-network-operator/iptables-alerter-dm7vf" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305482 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/853552d8-4997-4150-a85f-ac633b3fa6bd-konnectivity-ca\") pod \"konnectivity-agent-5hxt6\" (UID: \"853552d8-4997-4150-a85f-ac633b3fa6bd\") " pod="kube-system/konnectivity-agent-5hxt6" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305509 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d303bada-b544-4a08-936e-8b5b807d1628-cni-binary-copy\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305495 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-registration-dir\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305538 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305576 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9q8\" (UniqueName: \"kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8\") pod \"network-check-target-qcg9p\" (UID: \"1037cb0e-feaa-44fa-a256-9e9143945544\") " pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:28.305585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305585 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-var-lib-kubelet\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305607 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-run-netns\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305610 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d209e966-9242-4f9d-8d2d-63dc61ffc5c3-hosts-file\") pod \"node-resolver-r85xv\" (UID: \"d209e966-9242-4f9d-8d2d-63dc61ffc5c3\") " pod="openshift-dns/node-resolver-r85xv" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305641 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-node-log\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305659 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d209e966-9242-4f9d-8d2d-63dc61ffc5c3-hosts-file\") pod \"node-resolver-r85xv\" (UID: \"d209e966-9242-4f9d-8d2d-63dc61ffc5c3\") " pod="openshift-dns/node-resolver-r85xv" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305681 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-var-lib-kubelet\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305704 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-node-log\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305706 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-hostroot\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305736 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305762 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-sys\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305803 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-run-openvswitch\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305829 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e77adf8c-7131-49f1-b248-11a48c132170-env-overrides\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305832 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-sys\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305855 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szsf\" (UniqueName: \"kubernetes.io/projected/e77adf8c-7131-49f1-b248-11a48c132170-kube-api-access-4szsf\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305880 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-kubelet\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305907 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e77adf8c-7131-49f1-b248-11a48c132170-ovnkube-config\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305914 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-run-openvswitch\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305932 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e77adf8c-7131-49f1-b248-11a48c132170-ovn-node-metrics-cert\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.306328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305959 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/853552d8-4997-4150-a85f-ac633b3fa6bd-agent-certs\") pod \"konnectivity-agent-5hxt6\" (UID: \"853552d8-4997-4150-a85f-ac633b3fa6bd\") " pod="kube-system/konnectivity-agent-5hxt6" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305982 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-kubelet\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.305988 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac2f5397-4638-4694-9857-e5a25f186318-cnibin\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306043 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-sys-fs\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306068 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb-host\") pod \"node-ca-4mzss\" (UID: \"6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb\") " pod="openshift-image-registry/node-ca-4mzss" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306094 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e182034-0cd8-47b8-9c49-c872d2053550-host-slash\") pod \"iptables-alerter-dm7vf\" (UID: \"6e182034-0cd8-47b8-9c49-c872d2053550\") " pod="openshift-network-operator/iptables-alerter-dm7vf" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306147 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-etc-selinux\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306174 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-lib-modules\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306211 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-run-systemd\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306239 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-multus-socket-dir-parent\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306266 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6x8\" (UniqueName: \"kubernetes.io/projected/ac2f5397-4638-4694-9857-e5a25f186318-kube-api-access-jn6x8\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306296 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-device-dir\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306321 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-host\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306347 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d209e966-9242-4f9d-8d2d-63dc61ffc5c3-tmp-dir\") pod \"node-resolver-r85xv\" (UID: \"d209e966-9242-4f9d-8d2d-63dc61ffc5c3\") " pod="openshift-dns/node-resolver-r85xv" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306346 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e77adf8c-7131-49f1-b248-11a48c132170-env-overrides\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306390 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306424 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.307122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306423 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e182034-0cd8-47b8-9c49-c872d2053550-host-slash\") pod \"iptables-alerter-dm7vf\" (UID: \"6e182034-0cd8-47b8-9c49-c872d2053550\") " pod="openshift-network-operator/iptables-alerter-dm7vf" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306455 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-system-cni-dir\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306478 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-cnibin\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306505 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac2f5397-4638-4694-9857-e5a25f186318-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306529 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-sysctl-d\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306534 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-etc-selinux\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306554 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-cni-bin\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306582 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-multus-conf-dir\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306608 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gj54\" (UniqueName: \"kubernetes.io/projected/d303bada-b544-4a08-936e-8b5b807d1628-kube-api-access-9gj54\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306624 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-lib-modules\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306634 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7jv9\" (UniqueName: \"kubernetes.io/projected/6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb-kube-api-access-n7jv9\") pod \"node-ca-4mzss\" (UID: \"6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb\") " pod="openshift-image-registry/node-ca-4mzss" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306671 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-run-systemd\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306690 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-sysctl-conf\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306716 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-var-lib-cni-bin\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306743 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f547eaf0-def0-4524-9755-e682cd106e33-dbus\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306769 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-sysconfig\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306795 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-log-socket\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.307936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306805 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ac2f5397-4638-4694-9857-e5a25f186318-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306837 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-var-lib-cni-multus\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306866 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac2f5397-4638-4694-9857-e5a25f186318-system-cni-dir\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306869 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-cni-bin\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306893 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vzn9\" (UniqueName: \"kubernetes.io/projected/6e182034-0cd8-47b8-9c49-c872d2053550-kube-api-access-4vzn9\") pod \"iptables-alerter-dm7vf\" (UID: \"6e182034-0cd8-47b8-9c49-c872d2053550\") " pod="openshift-network-operator/iptables-alerter-dm7vf" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306919 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-systemd\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306941 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-device-dir\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307075 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-sysctl-conf\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.306943 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-var-lib-openvswitch\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307230 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-sysconfig\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307250 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d209e966-9242-4f9d-8d2d-63dc61ffc5c3-tmp-dir\") pod \"node-resolver-r85xv\" (UID: \"d209e966-9242-4f9d-8d2d-63dc61ffc5c3\") " pod="openshift-dns/node-resolver-r85xv" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307263 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79384694-7eec-4e99-a246-0799138642be-tmp\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307277 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307296 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-etc-openvswitch\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307318 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307335 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac2f5397-4638-4694-9857-e5a25f186318-cnibin\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307338 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-run-ovn\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.308801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307369 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-sys-fs\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307375 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-cni-netd\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307419 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e77adf8c-7131-49f1-b248-11a48c132170-ovnkube-script-lib\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307464 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-multus-cni-dir\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307491 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-run-netns\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307506 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-host\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307517 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d303bada-b544-4a08-936e-8b5b807d1628-multus-daemon-config\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307466 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-sysctl-d\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307613 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-log-socket\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307652 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac2f5397-4638-4694-9857-e5a25f186318-system-cni-dir\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307687 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-etc-openvswitch\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307726 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e77adf8c-7131-49f1-b248-11a48c132170-ovnkube-config\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307773 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-etc-systemd\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307812 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-var-lib-openvswitch\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307810 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-run\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307839 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ac2f5397-4638-4694-9857-e5a25f186318-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307855 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.309658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307885 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac2f5397-4638-4694-9857-e5a25f186318-cni-binary-copy\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.310398 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307908 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-host-cni-netd\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.310398 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307913 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac2f5397-4638-4694-9857-e5a25f186318-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.310398 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.307988 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79384694-7eec-4e99-a246-0799138642be-run\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.310398 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.308011 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e77adf8c-7131-49f1-b248-11a48c132170-run-ovn\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.310398 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.308051 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac2f5397-4638-4694-9857-e5a25f186318-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.310398 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.308060 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c7b0712-3366-4635-b6b8-386c025c3da3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.310398 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.308447 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac2f5397-4638-4694-9857-e5a25f186318-cni-binary-copy\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.310398 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.308726 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e77adf8c-7131-49f1-b248-11a48c132170-ovnkube-script-lib\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.310398 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.310240 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79384694-7eec-4e99-a246-0799138642be-etc-tuned\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.310739 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.310623 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79384694-7eec-4e99-a246-0799138642be-tmp\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.310972 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.310950 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e77adf8c-7131-49f1-b248-11a48c132170-ovn-node-metrics-cert\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.318469 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.318371 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k76rk\" (UniqueName: \"kubernetes.io/projected/79384694-7eec-4e99-a246-0799138642be-kube-api-access-k76rk\") pod \"tuned-pzmbs\" (UID: \"79384694-7eec-4e99-a246-0799138642be\") " pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.319137 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.319093 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmpdp\" (UniqueName: \"kubernetes.io/projected/d209e966-9242-4f9d-8d2d-63dc61ffc5c3-kube-api-access-cmpdp\") pod \"node-resolver-r85xv\" (UID: \"d209e966-9242-4f9d-8d2d-63dc61ffc5c3\") " pod="openshift-dns/node-resolver-r85xv" Sep 11 15:08:28.319244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.319222 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9f7s\" (UniqueName: \"kubernetes.io/projected/1c7b0712-3366-4635-b6b8-386c025c3da3-kube-api-access-r9f7s\") pod \"aws-ebs-csi-driver-node-qmmmj\" (UID: \"1c7b0712-3366-4635-b6b8-386c025c3da3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.319398 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.319373 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szsf\" (UniqueName: \"kubernetes.io/projected/e77adf8c-7131-49f1-b248-11a48c132170-kube-api-access-4szsf\") pod \"ovnkube-node-jbxnk\" (UID: \"e77adf8c-7131-49f1-b248-11a48c132170\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.319610 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.319417 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vzn9\" (UniqueName: \"kubernetes.io/projected/6e182034-0cd8-47b8-9c49-c872d2053550-kube-api-access-4vzn9\") pod \"iptables-alerter-dm7vf\" (UID: \"6e182034-0cd8-47b8-9c49-c872d2053550\") " pod="openshift-network-operator/iptables-alerter-dm7vf" Sep 11 15:08:28.319666 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.319653 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6x8\" (UniqueName: \"kubernetes.io/projected/ac2f5397-4638-4694-9857-e5a25f186318-kube-api-access-jn6x8\") pod \"multus-additional-cni-plugins-6b84v\" (UID: \"ac2f5397-4638-4694-9857-e5a25f186318\") " pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.408870 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.408832 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-system-cni-dir\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.408870 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.408873 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-cnibin\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.408893 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-multus-conf-dir\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.408913 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gj54\" (UniqueName: \"kubernetes.io/projected/d303bada-b544-4a08-936e-8b5b807d1628-kube-api-access-9gj54\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.408937 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7jv9\" (UniqueName: \"kubernetes.io/projected/6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb-kube-api-access-n7jv9\") pod \"node-ca-4mzss\" (UID: \"6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb\") " pod="openshift-image-registry/node-ca-4mzss" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.408963 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-var-lib-cni-bin\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.408987 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f547eaf0-def0-4524-9755-e682cd106e33-dbus\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.408988 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-multus-conf-dir\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409003 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-cnibin\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409012 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-var-lib-cni-multus\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409046 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-var-lib-cni-bin\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409050 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-var-lib-cni-multus\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.408987 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-system-cni-dir\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409080 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-multus-cni-dir\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409109 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-run-netns\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409136 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d303bada-b544-4a08-936e-8b5b807d1628-multus-daemon-config\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409173 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f547eaf0-def0-4524-9755-e682cd106e33-dbus\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409177 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-run-multus-certs\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409219 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-multus-cni-dir\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409222 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dz6w\" (UniqueName: \"kubernetes.io/projected/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-kube-api-access-9dz6w\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409223 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-run-multus-certs\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409275 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f547eaf0-def0-4524-9755-e682cd106e33-kubelet-config\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409321 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f547eaf0-def0-4524-9755-e682cd106e33-kubelet-config\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409331 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb-serviceca\") pod \"node-ca-4mzss\" (UID: \"6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb\") " pod="openshift-image-registry/node-ca-4mzss" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409358 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-run-netns\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409375 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-etc-kubernetes\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409408 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-run-k8s-cni-cncf-io\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409435 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-etc-kubernetes\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409473 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-run-k8s-cni-cncf-io\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409440 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/853552d8-4997-4150-a85f-ac633b3fa6bd-konnectivity-ca\") pod \"konnectivity-agent-5hxt6\" (UID: \"853552d8-4997-4150-a85f-ac633b3fa6bd\") " pod="kube-system/konnectivity-agent-5hxt6" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409521 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d303bada-b544-4a08-936e-8b5b807d1628-cni-binary-copy\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409552 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409581 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9q8\" (UniqueName: \"kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8\") pod \"network-check-target-qcg9p\" (UID: \"1037cb0e-feaa-44fa-a256-9e9143945544\") " pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:28.409737 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409612 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-var-lib-kubelet\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409639 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-hostroot\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409663 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409695 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/853552d8-4997-4150-a85f-ac633b3fa6bd-agent-certs\") pod \"konnectivity-agent-5hxt6\" (UID: \"853552d8-4997-4150-a85f-ac633b3fa6bd\") " pod="kube-system/konnectivity-agent-5hxt6" Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409720 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb-host\") pod \"node-ca-4mzss\" (UID: \"6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb\") " pod="openshift-image-registry/node-ca-4mzss" Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409743 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d303bada-b544-4a08-936e-8b5b807d1628-multus-daemon-config\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409750 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-multus-socket-dir-parent\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409800 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb-serviceca\") pod \"node-ca-4mzss\" (UID: \"6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb\") " pod="openshift-image-registry/node-ca-4mzss" Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409801 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-host-var-lib-kubelet\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.409857 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-multus-socket-dir-parent\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.409889 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.409978 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs podName:aaf8fe6d-2de1-4a11-9798-5d3e2e83164e nodeName:}" failed. No retries permitted until 2026-09-11 15:08:28.909957082 +0000 UTC m=+3.141881511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs") pod "network-metrics-daemon-lkx5k" (UID: "aaf8fe6d-2de1-4a11-9798-5d3e2e83164e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.410028 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/853552d8-4997-4150-a85f-ac633b3fa6bd-konnectivity-ca\") pod \"konnectivity-agent-5hxt6\" (UID: \"853552d8-4997-4150-a85f-ac633b3fa6bd\") " pod="kube-system/konnectivity-agent-5hxt6" Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.410094 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb-host\") pod \"node-ca-4mzss\" (UID: \"6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb\") " pod="openshift-image-registry/node-ca-4mzss" Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.410149 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d303bada-b544-4a08-936e-8b5b807d1628-hostroot\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.410171 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.410235 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret podName:f547eaf0-def0-4524-9755-e682cd106e33 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:28.910216066 +0000 UTC m=+3.142140481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret") pod "global-pull-secret-syncer-cdnlq" (UID: "f547eaf0-def0-4524-9755-e682cd106e33") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:28.410588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.410337 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d303bada-b544-4a08-936e-8b5b807d1628-cni-binary-copy\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.412801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.412768 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/853552d8-4997-4150-a85f-ac633b3fa6bd-agent-certs\") pod \"konnectivity-agent-5hxt6\" (UID: \"853552d8-4997-4150-a85f-ac633b3fa6bd\") " pod="kube-system/konnectivity-agent-5hxt6" Sep 11 15:08:28.418854 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.418788 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:08:28.418854 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.418811 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:08:28.418854 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.418826 2558 projected.go:196] Error preparing data for projected volume kube-api-access-bz9q8 for pod openshift-network-diagnostics/network-check-target-qcg9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:28.419070 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.418903 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8 podName:1037cb0e-feaa-44fa-a256-9e9143945544 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:28.918886098 +0000 UTC m=+3.150810509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bz9q8" (UniqueName: "kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8") pod "network-check-target-qcg9p" (UID: "1037cb0e-feaa-44fa-a256-9e9143945544") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:28.423992 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.423943 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7jv9\" (UniqueName: \"kubernetes.io/projected/6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb-kube-api-access-n7jv9\") pod \"node-ca-4mzss\" (UID: \"6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb\") " pod="openshift-image-registry/node-ca-4mzss" Sep 11 15:08:28.424647 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.424625 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gj54\" (UniqueName: \"kubernetes.io/projected/d303bada-b544-4a08-936e-8b5b807d1628-kube-api-access-9gj54\") pod \"multus-skzwk\" (UID: \"d303bada-b544-4a08-936e-8b5b807d1628\") " pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.425967 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.425944 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dz6w\" (UniqueName: \"kubernetes.io/projected/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-kube-api-access-9dz6w\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:28.491734 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.491684 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6b84v" Sep 11 15:08:28.502330 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.502302 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dm7vf" Sep 11 15:08:28.504494 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:28.504453 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2f5397_4638_4694_9857_e5a25f186318.slice/crio-0a2e7eddc89381bec0eec5c1216fbaa4f5596cc5ea9d4ed6b1aadd03ce438c1f WatchSource:0}: Error finding container 0a2e7eddc89381bec0eec5c1216fbaa4f5596cc5ea9d4ed6b1aadd03ce438c1f: Status 404 returned error can't find the container with id 0a2e7eddc89381bec0eec5c1216fbaa4f5596cc5ea9d4ed6b1aadd03ce438c1f Sep 11 15:08:28.511942 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.511917 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" Sep 11 15:08:28.513374 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:28.513347 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e182034_0cd8_47b8_9c49_c872d2053550.slice/crio-a99507df2682b04c300fcd83d96a7eb4346aeb2da32bdfa9eecfeed6560571a1 WatchSource:0}: Error finding container a99507df2682b04c300fcd83d96a7eb4346aeb2da32bdfa9eecfeed6560571a1: Status 404 returned error can't find the container with id a99507df2682b04c300fcd83d96a7eb4346aeb2da32bdfa9eecfeed6560571a1 Sep 11 15:08:28.520744 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.520720 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" Sep 11 15:08:28.528795 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.528773 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r85xv" Sep 11 15:08:28.537634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.537610 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:28.550463 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.550433 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5hxt6" Sep 11 15:08:28.557619 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.557597 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-skzwk" Sep 11 15:08:28.563692 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.563664 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4mzss" Sep 11 15:08:28.915248 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.915160 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:28.915248 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:28.915210 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:28.915455 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.915324 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:28.915455 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.915345 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:28.915455 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.915388 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs podName:aaf8fe6d-2de1-4a11-9798-5d3e2e83164e nodeName:}" failed. No retries permitted until 2026-09-11 15:08:29.915373244 +0000 UTC m=+4.147297654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs") pod "network-metrics-daemon-lkx5k" (UID: "aaf8fe6d-2de1-4a11-9798-5d3e2e83164e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:28.915455 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:28.915417 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret podName:f547eaf0-def0-4524-9755-e682cd106e33 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:29.915394747 +0000 UTC m=+4.147319161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret") pod "global-pull-secret-syncer-cdnlq" (UID: "f547eaf0-def0-4524-9755-e682cd106e33") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:29.016143 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:29.016105 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9q8\" (UniqueName: \"kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8\") pod \"network-check-target-qcg9p\" (UID: \"1037cb0e-feaa-44fa-a256-9e9143945544\") " pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:29.016314 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:29.016292 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:08:29.016372 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:29.016322 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:08:29.016372 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:29.016333 2558 projected.go:196] Error preparing data for projected volume kube-api-access-bz9q8 for pod openshift-network-diagnostics/network-check-target-qcg9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:29.016461 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:29.016401 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8 podName:1037cb0e-feaa-44fa-a256-9e9143945544 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:30.016379676 +0000 UTC m=+4.248304104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bz9q8" (UniqueName: "kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8") pod "network-check-target-qcg9p" (UID: "1037cb0e-feaa-44fa-a256-9e9143945544") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:29.241653 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:29.241583 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 15:03:28 +0000 UTC" deadline="2028-06-07 07:25:23.081793496 +0000 UTC" Sep 11 15:08:29.241653 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:29.241611 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15232h16m53.840185845s" Sep 11 15:08:29.268804 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:29.268746 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b84v" event={"ID":"ac2f5397-4638-4694-9857-e5a25f186318","Type":"ContainerStarted","Data":"0a2e7eddc89381bec0eec5c1216fbaa4f5596cc5ea9d4ed6b1aadd03ce438c1f"} Sep 11 15:08:29.269908 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:29.269879 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dm7vf" event={"ID":"6e182034-0cd8-47b8-9c49-c872d2053550","Type":"ContainerStarted","Data":"a99507df2682b04c300fcd83d96a7eb4346aeb2da32bdfa9eecfeed6560571a1"} Sep 11 15:08:29.651562 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:29.651525 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7b0712_3366_4635_b6b8_386c025c3da3.slice/crio-b6b1f2bde97d9418aec0ff283bbbc04ae19474c5b66d955a31eab3c72af9e1ef WatchSource:0}: Error finding container b6b1f2bde97d9418aec0ff283bbbc04ae19474c5b66d955a31eab3c72af9e1ef: Status 404 returned error can't find the container with id b6b1f2bde97d9418aec0ff283bbbc04ae19474c5b66d955a31eab3c72af9e1ef Sep 11 15:08:29.658885 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:29.658777 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd303bada_b544_4a08_936e_8b5b807d1628.slice/crio-be48c340a0778c5cf17bf22a602563faca7ca57b10df9acd70f755a5532bf90f WatchSource:0}: Error finding container be48c340a0778c5cf17bf22a602563faca7ca57b10df9acd70f755a5532bf90f: Status 404 returned error can't find the container with id be48c340a0778c5cf17bf22a602563faca7ca57b10df9acd70f755a5532bf90f Sep 11 15:08:29.661282 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:29.661113 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd209e966_9242_4f9d_8d2d_63dc61ffc5c3.slice/crio-37cb6b5bb8eea683fed6e208124acc58e73b53e34ca164f0d5b024b944f7250d WatchSource:0}: Error finding container 37cb6b5bb8eea683fed6e208124acc58e73b53e34ca164f0d5b024b944f7250d: Status 404 returned error can't find the container with id 37cb6b5bb8eea683fed6e208124acc58e73b53e34ca164f0d5b024b944f7250d Sep 11 15:08:29.662090 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:29.661650 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853552d8_4997_4150_a85f_ac633b3fa6bd.slice/crio-1a3244544f3152d74706369281c0e72981c36bbc09a8e8edca3eefb379394ce0 WatchSource:0}: Error finding container 1a3244544f3152d74706369281c0e72981c36bbc09a8e8edca3eefb379394ce0: Status 404 returned error can't find the container with id 1a3244544f3152d74706369281c0e72981c36bbc09a8e8edca3eefb379394ce0 Sep 11 15:08:29.662292 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:29.662119 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddcf6d3_17a1_4b8a_b563_dd7be8640bcb.slice/crio-9cc20a08abf3fec0a4f036f0d23828040f7afc4da5973ffa685665644415827c WatchSource:0}: Error finding container 9cc20a08abf3fec0a4f036f0d23828040f7afc4da5973ffa685665644415827c: Status 404 returned error can't find the container with id 9cc20a08abf3fec0a4f036f0d23828040f7afc4da5973ffa685665644415827c Sep 11 15:08:29.663524 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:29.663501 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79384694_7eec_4e99_a246_0799138642be.slice/crio-a92400ba08870d479cef4796c54db001af91beda8b3de88eb9e6d14cafc7babd WatchSource:0}: Error finding container a92400ba08870d479cef4796c54db001af91beda8b3de88eb9e6d14cafc7babd: Status 404 returned error can't find the container with id a92400ba08870d479cef4796c54db001af91beda8b3de88eb9e6d14cafc7babd Sep 11 15:08:29.665666 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:08:29.665640 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77adf8c_7131_49f1_b248_11a48c132170.slice/crio-fd2b0113761986643ed79ee25ad2c2869afaf15adb8a13f61cb3902a55a58939 WatchSource:0}: Error finding container fd2b0113761986643ed79ee25ad2c2869afaf15adb8a13f61cb3902a55a58939: Status 404 returned error can't find the container with id fd2b0113761986643ed79ee25ad2c2869afaf15adb8a13f61cb3902a55a58939 Sep 11 15:08:29.925068 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:29.924976 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:29.925232 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:29.925176 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:29.925304 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:29.925282 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:29.925365 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:29.925286 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret podName:f547eaf0-def0-4524-9755-e682cd106e33 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:31.925243921 +0000 UTC m=+6.157168333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret") pod "global-pull-secret-syncer-cdnlq" (UID: "f547eaf0-def0-4524-9755-e682cd106e33") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:29.925434 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:29.925363 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs podName:aaf8fe6d-2de1-4a11-9798-5d3e2e83164e nodeName:}" failed. No retries permitted until 2026-09-11 15:08:31.925340008 +0000 UTC m=+6.157264420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs") pod "network-metrics-daemon-lkx5k" (UID: "aaf8fe6d-2de1-4a11-9798-5d3e2e83164e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:29.925434 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:29.925151 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:30.026428 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.026390 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9q8\" (UniqueName: \"kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8\") pod \"network-check-target-qcg9p\" (UID: \"1037cb0e-feaa-44fa-a256-9e9143945544\") " pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:30.026633 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:30.026605 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:08:30.026757 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:30.026638 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:08:30.026757 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:30.026654 2558 projected.go:196] Error preparing data for projected volume kube-api-access-bz9q8 for pod openshift-network-diagnostics/network-check-target-qcg9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:30.026757 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:30.026727 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8 podName:1037cb0e-feaa-44fa-a256-9e9143945544 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:32.026704882 +0000 UTC m=+6.258629311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bz9q8" (UniqueName: "kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8") pod "network-check-target-qcg9p" (UID: "1037cb0e-feaa-44fa-a256-9e9143945544") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:30.242632 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.242512 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 15:03:28 +0000 UTC" deadline="2028-06-15 04:28:29.76276812 +0000 UTC" Sep 11 15:08:30.242632 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.242583 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15421h19m59.520190612s" Sep 11 15:08:30.262772 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.262516 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:30.262935 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.262628 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:30.262935 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:30.262893 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:30.263046 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:30.262968 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:30.263046 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.262720 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:30.263137 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:30.263079 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:30.274154 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.274124 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-skzwk" event={"ID":"d303bada-b544-4a08-936e-8b5b807d1628","Type":"ContainerStarted","Data":"be48c340a0778c5cf17bf22a602563faca7ca57b10df9acd70f755a5532bf90f"} Sep 11 15:08:30.277610 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.277583 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" event={"ID":"4e1779d60c22f5ff6cb06e66ddefda83","Type":"ContainerStarted","Data":"5f6d0c86b34cdc3d7129c860c94bd7669c889ad2b36f00e24ef2385b95b9257b"} Sep 11 15:08:30.279839 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.279792 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" event={"ID":"79384694-7eec-4e99-a246-0799138642be","Type":"ContainerStarted","Data":"a92400ba08870d479cef4796c54db001af91beda8b3de88eb9e6d14cafc7babd"} Sep 11 15:08:30.282465 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.282393 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" event={"ID":"1c7b0712-3366-4635-b6b8-386c025c3da3","Type":"ContainerStarted","Data":"b6b1f2bde97d9418aec0ff283bbbc04ae19474c5b66d955a31eab3c72af9e1ef"} Sep 11 15:08:30.286561 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.285945 2558 generic.go:356] "Generic (PLEG): container finished" podID="2ffa511d6fab2cc6c0e8967b38c06790" containerID="4c45bc31fd3b3fb73d4c56ad4cafc8e93397fab0a29256f924c60e711c4a47f7" exitCode=0 Sep 11 15:08:30.286561 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.286023 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" event={"ID":"2ffa511d6fab2cc6c0e8967b38c06790","Type":"ContainerDied","Data":"4c45bc31fd3b3fb73d4c56ad4cafc8e93397fab0a29256f924c60e711c4a47f7"} Sep 11 15:08:30.289537 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.289260 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" event={"ID":"e77adf8c-7131-49f1-b248-11a48c132170","Type":"ContainerStarted","Data":"fd2b0113761986643ed79ee25ad2c2869afaf15adb8a13f61cb3902a55a58939"} Sep 11 15:08:30.292508 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.292417 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4mzss" event={"ID":"6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb","Type":"ContainerStarted","Data":"9cc20a08abf3fec0a4f036f0d23828040f7afc4da5973ffa685665644415827c"} Sep 11 15:08:30.293973 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.293709 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-128.ec2.internal" podStartSLOduration=2.29369143 podStartE2EDuration="2.29369143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:30.293157594 +0000 UTC m=+4.525082032" watchObservedRunningTime="2026-09-11 15:08:30.29369143 +0000 UTC m=+4.525615866" Sep 11 15:08:30.294714 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.294506 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5hxt6" event={"ID":"853552d8-4997-4150-a85f-ac633b3fa6bd","Type":"ContainerStarted","Data":"1a3244544f3152d74706369281c0e72981c36bbc09a8e8edca3eefb379394ce0"} Sep 11 15:08:30.297499 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:30.297452 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r85xv" event={"ID":"d209e966-9242-4f9d-8d2d-63dc61ffc5c3","Type":"ContainerStarted","Data":"37cb6b5bb8eea683fed6e208124acc58e73b53e34ca164f0d5b024b944f7250d"} Sep 11 15:08:31.317978 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:31.317885 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" event={"ID":"2ffa511d6fab2cc6c0e8967b38c06790","Type":"ContainerStarted","Data":"c1824c50d1c517e807b59e29cb95268e53c6c70f813367c5225042973d516ff9"} Sep 11 15:08:31.335975 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:31.335917 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-128.ec2.internal" podStartSLOduration=3.335892634 podStartE2EDuration="3.335892634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:31.335408693 +0000 UTC m=+5.567333124" watchObservedRunningTime="2026-09-11 15:08:31.335892634 +0000 UTC m=+5.567817072" Sep 11 15:08:31.944973 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:31.944929 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:31.945218 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:31.945008 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:31.945218 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:31.945142 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:31.945353 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:31.945224 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs podName:aaf8fe6d-2de1-4a11-9798-5d3e2e83164e nodeName:}" failed. No retries permitted until 2026-09-11 15:08:35.945202761 +0000 UTC m=+10.177127190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs") pod "network-metrics-daemon-lkx5k" (UID: "aaf8fe6d-2de1-4a11-9798-5d3e2e83164e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:31.945353 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:31.945296 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:31.945353 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:31.945332 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret podName:f547eaf0-def0-4524-9755-e682cd106e33 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:35.945320817 +0000 UTC m=+10.177245228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret") pod "global-pull-secret-syncer-cdnlq" (UID: "f547eaf0-def0-4524-9755-e682cd106e33") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:32.045904 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:32.045857 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9q8\" (UniqueName: \"kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8\") pod \"network-check-target-qcg9p\" (UID: \"1037cb0e-feaa-44fa-a256-9e9143945544\") " pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:32.046144 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:32.046117 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:08:32.046227 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:32.046147 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:08:32.046227 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:32.046163 2558 projected.go:196] Error preparing data for projected volume kube-api-access-bz9q8 for pod openshift-network-diagnostics/network-check-target-qcg9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:32.046340 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:32.046228 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8 podName:1037cb0e-feaa-44fa-a256-9e9143945544 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:36.046208101 +0000 UTC m=+10.278132536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bz9q8" (UniqueName: "kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8") pod "network-check-target-qcg9p" (UID: "1037cb0e-feaa-44fa-a256-9e9143945544") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:32.260767 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:32.260591 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:32.260767 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:32.260750 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:32.261278 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:32.261160 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:32.261278 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:32.261264 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:32.261443 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:32.261311 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:32.261443 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:32.261380 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:34.259985 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:34.259948 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:34.260455 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:34.260003 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:34.260455 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:34.260114 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:34.260611 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:34.260581 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:34.260675 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:34.260645 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:34.260741 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:34.260721 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:35.981334 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:35.981288 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:35.981836 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:35.981357 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:35.981836 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:35.981489 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:35.981836 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:35.981494 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:35.981836 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:35.981568 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs podName:aaf8fe6d-2de1-4a11-9798-5d3e2e83164e nodeName:}" failed. No retries permitted until 2026-09-11 15:08:43.981551912 +0000 UTC m=+18.213476324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs") pod "network-metrics-daemon-lkx5k" (UID: "aaf8fe6d-2de1-4a11-9798-5d3e2e83164e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:35.981836 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:35.981584 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret podName:f547eaf0-def0-4524-9755-e682cd106e33 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:43.981577954 +0000 UTC m=+18.213502364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret") pod "global-pull-secret-syncer-cdnlq" (UID: "f547eaf0-def0-4524-9755-e682cd106e33") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:36.082124 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:36.081718 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9q8\" (UniqueName: \"kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8\") pod \"network-check-target-qcg9p\" (UID: \"1037cb0e-feaa-44fa-a256-9e9143945544\") " pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:36.082124 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:36.081998 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:08:36.082124 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:36.082022 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:08:36.082124 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:36.082036 2558 projected.go:196] Error preparing data for projected volume kube-api-access-bz9q8 for pod openshift-network-diagnostics/network-check-target-qcg9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:36.082124 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:36.082101 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8 podName:1037cb0e-feaa-44fa-a256-9e9143945544 nodeName:}" failed. No retries permitted until 2026-09-11 15:08:44.08208111 +0000 UTC m=+18.314005546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bz9q8" (UniqueName: "kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8") pod "network-check-target-qcg9p" (UID: "1037cb0e-feaa-44fa-a256-9e9143945544") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:36.259423 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:36.259326 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:36.259602 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:36.259476 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:36.259602 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:36.259591 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:36.259742 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:36.259624 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:36.259742 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:36.259717 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:36.260224 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:36.259954 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:38.258395 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:38.258362 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:38.258914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:38.258362 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:38.258914 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:38.258510 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:38.258914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:38.258606 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:38.258914 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:38.258596 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:38.259341 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:38.259177 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:40.258657 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:40.258629 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:40.259100 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:40.258775 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:40.259100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:40.259084 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:40.259217 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:40.259185 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:40.259269 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:40.259243 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:40.259316 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:40.259303 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:42.258448 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:42.258406 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:42.258448 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:42.258435 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:42.258954 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:42.258416 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:42.258954 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:42.258538 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:42.258954 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:42.258631 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:42.258954 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:42.258769 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:44.039287 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:44.039255 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:44.039672 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:44.039301 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:44.039672 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:44.039393 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:44.039672 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:44.039442 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs podName:aaf8fe6d-2de1-4a11-9798-5d3e2e83164e nodeName:}" failed. No retries permitted until 2026-09-11 15:09:00.039428921 +0000 UTC m=+34.271353330 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs") pod "network-metrics-daemon-lkx5k" (UID: "aaf8fe6d-2de1-4a11-9798-5d3e2e83164e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:08:44.039672 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:44.039393 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:44.039672 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:44.039512 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret podName:f547eaf0-def0-4524-9755-e682cd106e33 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:00.039499325 +0000 UTC m=+34.271423749 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret") pod "global-pull-secret-syncer-cdnlq" (UID: "f547eaf0-def0-4524-9755-e682cd106e33") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:08:44.140169 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:44.140097 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9q8\" (UniqueName: \"kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8\") pod \"network-check-target-qcg9p\" (UID: \"1037cb0e-feaa-44fa-a256-9e9143945544\") " pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:44.140321 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:44.140240 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:08:44.140321 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:44.140255 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:08:44.140321 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:44.140264 2558 projected.go:196] Error preparing data for projected volume kube-api-access-bz9q8 for pod openshift-network-diagnostics/network-check-target-qcg9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:44.140321 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:44.140318 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8 podName:1037cb0e-feaa-44fa-a256-9e9143945544 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:00.140301904 +0000 UTC m=+34.372226327 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bz9q8" (UniqueName: "kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8") pod "network-check-target-qcg9p" (UID: "1037cb0e-feaa-44fa-a256-9e9143945544") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:08:44.258479 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:44.258443 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:44.258665 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:44.258443 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:44.258665 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:44.258645 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:44.258808 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:44.258734 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:44.258866 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:44.258848 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:44.258932 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:44.258915 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:46.258654 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:46.258607 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:46.259127 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:46.258714 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:46.259127 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:46.258763 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:46.259127 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:46.258796 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:46.259127 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:46.258835 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:46.259127 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:46.258947 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:48.259092 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.258669 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:48.259938 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.258683 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:48.259938 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.258731 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:48.259938 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:48.259269 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:48.259938 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:48.259329 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:48.259938 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:48.259218 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:48.354208 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.354179 2558 generic.go:356] "Generic (PLEG): container finished" podID="ac2f5397-4638-4694-9857-e5a25f186318" containerID="28c95b2e6735b2c5f284d2062aee1dcd90a6e46efc8aea917bd318c0dd271d78" exitCode=0 Sep 11 15:08:48.354346 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.354269 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b84v" event={"ID":"ac2f5397-4638-4694-9857-e5a25f186318","Type":"ContainerDied","Data":"28c95b2e6735b2c5f284d2062aee1dcd90a6e46efc8aea917bd318c0dd271d78"} Sep 11 15:08:48.357727 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.357684 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" event={"ID":"e77adf8c-7131-49f1-b248-11a48c132170","Type":"ContainerStarted","Data":"f41042bd90480b9de756641eac8ffb1ce6c7ddf9924816271c1a7731a031afc6"} Sep 11 15:08:48.357862 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.357799 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" event={"ID":"e77adf8c-7131-49f1-b248-11a48c132170","Type":"ContainerStarted","Data":"015f28fc0b49fd6a3a08bb6cc9db1e23ba998ad08a7fa5d0bf2f635ef9484aed"} Sep 11 15:08:48.357927 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.357862 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" event={"ID":"e77adf8c-7131-49f1-b248-11a48c132170","Type":"ContainerStarted","Data":"264071e079fa82c745095460021961286d0beafcfac55005b81551b6f4664c82"} Sep 11 15:08:48.357927 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.357912 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" event={"ID":"e77adf8c-7131-49f1-b248-11a48c132170","Type":"ContainerStarted","Data":"2f1d93e9a4e6626a73e725ad718faa631ec1da1345fc3bd491c7ce4d3cb92e24"} Sep 11 15:08:48.358026 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.357976 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" event={"ID":"e77adf8c-7131-49f1-b248-11a48c132170","Type":"ContainerStarted","Data":"9bd887b7f58c22859c77829cd173928ced01c7f1c5e2e04961b699727855af74"} Sep 11 15:08:48.358079 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.358031 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" event={"ID":"e77adf8c-7131-49f1-b248-11a48c132170","Type":"ContainerStarted","Data":"99c993efcf0ec3a99dc4a362924656f295bb60b40d239a6ce0b1695f30731a45"} Sep 11 15:08:48.359242 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.359221 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4mzss" event={"ID":"6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb","Type":"ContainerStarted","Data":"b37c95f4c51d00b5f7fcea19603168ddc1487806c7cbcd35d94ca18be81c51a4"} Sep 11 15:08:48.363238 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.363211 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5hxt6" event={"ID":"853552d8-4997-4150-a85f-ac633b3fa6bd","Type":"ContainerStarted","Data":"503840aaf12a7a7c64d63d6a141c8ff6bd242aca5f9992172b51b8420f46678d"} Sep 11 15:08:48.364633 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.364615 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r85xv" event={"ID":"d209e966-9242-4f9d-8d2d-63dc61ffc5c3","Type":"ContainerStarted","Data":"a1a7086021c61242db87ada34147f34c18567498a88116e260f54bad17058159"} Sep 11 15:08:48.366030 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.366012 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-skzwk" event={"ID":"d303bada-b544-4a08-936e-8b5b807d1628","Type":"ContainerStarted","Data":"4b6f6b1e8a8b91c9404939ca3c7f1772d9199303d24606ce97a308ea8c19d877"} Sep 11 15:08:48.367379 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.367362 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" event={"ID":"79384694-7eec-4e99-a246-0799138642be","Type":"ContainerStarted","Data":"ca539f469c0da5905915809a41e2d0e7bf18f7586de2ca43ae52e89515a62282"} Sep 11 15:08:48.368626 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.368609 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" event={"ID":"1c7b0712-3366-4635-b6b8-386c025c3da3","Type":"ContainerStarted","Data":"aa88a6a2a243f08660e4d0211df60d600a5165e5f44818496c075d6aae0f06bc"} Sep 11 15:08:48.405583 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.405500 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r85xv" podStartSLOduration=7.225137912 podStartE2EDuration="21.405488805s" totalImagesPullingTime="14.180350893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:48.389887308 +0000 UTC m=+22.621811738" watchObservedRunningTime="2026-09-11 15:08:48.405488805 +0000 UTC m=+22.637413217" Sep 11 15:08:48.405764 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.405730 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4mzss" podStartSLOduration=3.7014437669999998 podStartE2EDuration="21.405725245s" totalImagesPullingTime="17.704281478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:48.405475575 +0000 UTC m=+22.637400006" watchObservedRunningTime="2026-09-11 15:08:48.405725245 +0000 UTC m=+22.637649677" Sep 11 15:08:48.436438 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.436408 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5hxt6" podStartSLOduration=3.502320433 podStartE2EDuration="21.436396916s" totalImagesPullingTime="17.934076483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:48.43623719 +0000 UTC m=+22.668161621" watchObservedRunningTime="2026-09-11 15:08:48.436396916 +0000 UTC m=+22.668321348" Sep 11 15:08:48.436544 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:48.436527 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pzmbs" podStartSLOduration=3.502357572 podStartE2EDuration="21.436521458s" totalImagesPullingTime="17.934163886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:48.421354892 +0000 UTC m=+22.653279323" watchObservedRunningTime="2026-09-11 15:08:48.436521458 +0000 UTC m=+22.668445944" Sep 11 15:08:49.202026 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:49.202001 2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 15:08:49.270082 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:49.269994 2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T15:08:49.202021677Z","UUID":"98b118c2-9f0f-4c31-b1a0-e1b3e5c52994","Handler":null,"Name":"","Endpoint":""} Sep 11 15:08:49.272322 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:49.272303 2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 15:08:49.272401 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:49.272334 2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 15:08:49.372782 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:49.372688 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dm7vf" event={"ID":"6e182034-0cd8-47b8-9c49-c872d2053550","Type":"ContainerStarted","Data":"865a145bc023b59262b20de608cf0fe9fce00748510c54687bcb068e635ebc5e"} Sep 11 15:08:49.374869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:49.374838 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" event={"ID":"1c7b0712-3366-4635-b6b8-386c025c3da3","Type":"ContainerStarted","Data":"95dea2f99132836d61343f4d39a31956b52f9229df9b0851d4869466564cc6ba"} Sep 11 15:08:49.386831 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:49.386789 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dm7vf" podStartSLOduration=7.05849753 podStartE2EDuration="22.386776098s" totalImagesPullingTime="15.328278568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:49.386771651 +0000 UTC m=+23.618696082" watchObservedRunningTime="2026-09-11 15:08:49.386776098 +0000 UTC m=+23.618700530" Sep 11 15:08:49.387339 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:49.387312 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-skzwk" podStartSLOduration=4.409639344 podStartE2EDuration="22.387301238s" totalImagesPullingTime="17.977661894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:48.456126625 +0000 UTC m=+22.688051056" watchObservedRunningTime="2026-09-11 15:08:49.387301238 +0000 UTC m=+23.619225669" Sep 11 15:08:50.054231 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:50.054168 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5hxt6" Sep 11 15:08:50.258507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:50.258442 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:50.258710 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:50.258684 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:50.259772 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:50.259658 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:50.260149 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:50.260122 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:50.260349 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:50.260325 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:50.260683 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:50.260659 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:51.382615 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:51.382532 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" event={"ID":"1c7b0712-3366-4635-b6b8-386c025c3da3","Type":"ContainerStarted","Data":"0e6de1c32c073cb1d96bf5d1ff576fea8f4d2f44f759b8ab83b26bb6ec4c809e"} Sep 11 15:08:51.386916 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:51.386664 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" event={"ID":"e77adf8c-7131-49f1-b248-11a48c132170","Type":"ContainerStarted","Data":"344ef094a15502bb86d154365e51d84a0c6e87f8992258f321842fe22500bbcf"} Sep 11 15:08:51.402961 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:51.402914 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qmmmj" podStartSLOduration=3.727408686 podStartE2EDuration="24.402900217s" totalImagesPullingTime="20.675491531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:51.402528822 +0000 UTC m=+25.634453257" watchObservedRunningTime="2026-09-11 15:08:51.402900217 +0000 UTC m=+25.634824648" Sep 11 15:08:52.258332 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:52.258299 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:52.258503 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:52.258438 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:52.258503 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:52.258458 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:52.258629 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:52.258299 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:52.258629 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:52.258571 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:52.258730 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:52.258638 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:52.552591 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:52.552511 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5hxt6" Sep 11 15:08:52.553161 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:52.553144 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5hxt6" Sep 11 15:08:53.392595 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:53.392565 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5hxt6" Sep 11 15:08:54.257901 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:54.257765 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:54.257901 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:54.257800 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:54.257901 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:54.257808 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:54.258543 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:54.257895 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:54.258543 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:54.257997 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:54.258543 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:54.258064 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:54.396223 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:54.395980 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b84v" event={"ID":"ac2f5397-4638-4694-9857-e5a25f186318","Type":"ContainerStarted","Data":"6938911d0a3d897fc80211c2e219e87166327deaaf8be20559d43bcce7671692"} Sep 11 15:08:54.400094 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:54.400017 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" event={"ID":"e77adf8c-7131-49f1-b248-11a48c132170","Type":"ContainerStarted","Data":"6f1e0e8798a1c089bbe92fd8992d2b5291028d68bb4711179789a5c4c7a9ecba"} Sep 11 15:08:54.441008 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:54.440962 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" podStartSLOduration=9.46220485 podStartE2EDuration="27.440946877s" totalImagesPullingTime="17.978742027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:08:54.439410192 +0000 UTC m=+28.671334624" watchObservedRunningTime="2026-09-11 15:08:54.440946877 +0000 UTC m=+28.672871357" Sep 11 15:08:55.404581 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.404547 2558 generic.go:356] "Generic (PLEG): container finished" podID="ac2f5397-4638-4694-9857-e5a25f186318" containerID="6938911d0a3d897fc80211c2e219e87166327deaaf8be20559d43bcce7671692" exitCode=0 Sep 11 15:08:55.405036 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.404620 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b84v" event={"ID":"ac2f5397-4638-4694-9857-e5a25f186318","Type":"ContainerDied","Data":"6938911d0a3d897fc80211c2e219e87166327deaaf8be20559d43bcce7671692"} Sep 11 15:08:55.405036 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.404894 2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 15:08:55.405931 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.405406 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:55.405931 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.405471 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:55.422125 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.422103 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:55.422438 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.422423 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:55.839244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.839161 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cdnlq"] Sep 11 15:08:55.839463 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.839306 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:55.839463 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:55.839426 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:55.842417 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.842392 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lkx5k"] Sep 11 15:08:55.842555 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.842495 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:55.842611 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:55.842595 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:55.843174 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.843151 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qcg9p"] Sep 11 15:08:55.843272 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.843245 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:55.843349 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:55.843328 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:55.992442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:55.992412 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:08:57.257780 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:57.257690 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:57.257780 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:57.257694 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:57.258155 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:57.257815 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:57.258155 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:57.257936 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:57.410260 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:57.410222 2558 generic.go:356] "Generic (PLEG): container finished" podID="ac2f5397-4638-4694-9857-e5a25f186318" containerID="4ad1086350e31c8c961edd7a344700adad9846a5d78549108843294421612dfd" exitCode=0 Sep 11 15:08:57.410411 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:57.410310 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b84v" event={"ID":"ac2f5397-4638-4694-9857-e5a25f186318","Type":"ContainerDied","Data":"4ad1086350e31c8c961edd7a344700adad9846a5d78549108843294421612dfd"} Sep 11 15:08:58.258480 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:58.258413 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:08:58.258824 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:58.258515 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:08:59.258615 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:59.258428 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:08:59.259008 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:59.258463 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:08:59.259008 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:59.258704 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkx5k" podUID="aaf8fe6d-2de1-4a11-9798-5d3e2e83164e" Sep 11 15:08:59.259008 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:08:59.258770 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcg9p" podUID="1037cb0e-feaa-44fa-a256-9e9143945544" Sep 11 15:08:59.416403 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:59.416373 2558 generic.go:356] "Generic (PLEG): container finished" podID="ac2f5397-4638-4694-9857-e5a25f186318" containerID="f82b38819befff0f461e22f48577a7aa57fcc90b779e38f4071bf188d0455cfd" exitCode=0 Sep 11 15:08:59.416568 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:08:59.416426 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b84v" event={"ID":"ac2f5397-4638-4694-9857-e5a25f186318","Type":"ContainerDied","Data":"f82b38819befff0f461e22f48577a7aa57fcc90b779e38f4071bf188d0455cfd"} Sep 11 15:09:00.055523 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.055496 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:09:00.055691 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.055539 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:09:00.055691 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.055651 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:09:00.055776 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.055711 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret podName:f547eaf0-def0-4524-9755-e682cd106e33 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:32.055695933 +0000 UTC m=+66.287620347 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret") pod "global-pull-secret-syncer-cdnlq" (UID: "f547eaf0-def0-4524-9755-e682cd106e33") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:09:00.055776 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.055712 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:09:00.055851 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.055777 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs podName:aaf8fe6d-2de1-4a11-9798-5d3e2e83164e nodeName:}" failed. No retries permitted until 2026-09-11 15:09:32.055764412 +0000 UTC m=+66.287688841 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs") pod "network-metrics-daemon-lkx5k" (UID: "aaf8fe6d-2de1-4a11-9798-5d3e2e83164e") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:09:00.156679 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.156655 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9q8\" (UniqueName: \"kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8\") pod \"network-check-target-qcg9p\" (UID: \"1037cb0e-feaa-44fa-a256-9e9143945544\") " pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:09:00.156839 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.156824 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:09:00.156882 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.156842 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:09:00.156882 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.156852 2558 projected.go:196] Error preparing data for projected volume kube-api-access-bz9q8 for pod openshift-network-diagnostics/network-check-target-qcg9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:09:00.156948 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.156898 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8 podName:1037cb0e-feaa-44fa-a256-9e9143945544 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:32.156883631 +0000 UTC m=+66.388808046 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bz9q8" (UniqueName: "kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8") pod "network-check-target-qcg9p" (UID: "1037cb0e-feaa-44fa-a256-9e9143945544") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:09:00.258208 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.258181 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:09:00.258328 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.258305 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cdnlq" podUID="f547eaf0-def0-4524-9755-e682cd106e33" Sep 11 15:09:00.551726 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.551700 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-128.ec2.internal" event="NodeReady" Sep 11 15:09:00.552062 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.551853 2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 15:09:00.586033 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.586009 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fddbb4949-qnxrp"] Sep 11 15:09:00.602406 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.602371 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tngb6"] Sep 11 15:09:00.603252 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.602613 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.604794 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.604771 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 15:09:00.605020 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.604996 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tz2qn\"" Sep 11 15:09:00.605270 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.605253 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 15:09:00.605478 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.605464 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 15:09:00.610299 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.610073 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 15:09:00.621239 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.621216 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fddbb4949-qnxrp"] Sep 11 15:09:00.621336 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.621249 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dp7jn"] Sep 11 15:09:00.621336 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.621294 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:00.623588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.623571 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 15:09:00.623677 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.623646 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zldr7\"" Sep 11 15:09:00.623816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.623803 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 15:09:00.634046 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.634028 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tngb6"] Sep 11 15:09:00.634131 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.634060 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dp7jn"] Sep 11 15:09:00.634168 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.634131 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dp7jn" Sep 11 15:09:00.636250 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.636233 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 15:09:00.636476 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.636461 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 15:09:00.636528 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.636494 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 15:09:00.636582 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.636565 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6rfh2\"" Sep 11 15:09:00.761241 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761207 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-bound-sa-token\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.761396 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761270 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20d09616-cdc7-41d0-9a02-f888fb741327-registry-certificates\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.761396 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761294 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20d09616-cdc7-41d0-9a02-f888fb741327-trusted-ca\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.761396 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761311 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20d09616-cdc7-41d0-9a02-f888fb741327-installation-pull-secrets\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.761396 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761332 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-config-volume\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:00.761396 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761378 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwn98\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-kube-api-access-pwn98\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.761606 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761398 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-tmp-dir\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:00.761606 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761443 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20d09616-cdc7-41d0-9a02-f888fb741327-ca-trust-extracted\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.761606 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761487 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbgsv\" (UniqueName: \"kubernetes.io/projected/379aa0e8-9d7d-42dc-bd0d-acb425c71a2d-kube-api-access-zbgsv\") pod \"ingress-canary-dp7jn\" (UID: \"379aa0e8-9d7d-42dc-bd0d-acb425c71a2d\") " pod="openshift-ingress-canary/ingress-canary-dp7jn" Sep 11 15:09:00.761606 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761521 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.761606 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761540 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/379aa0e8-9d7d-42dc-bd0d-acb425c71a2d-cert\") pod \"ingress-canary-dp7jn\" (UID: \"379aa0e8-9d7d-42dc-bd0d-acb425c71a2d\") " pod="openshift-ingress-canary/ingress-canary-dp7jn" Sep 11 15:09:00.761606 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761589 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:00.761903 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761633 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20d09616-cdc7-41d0-9a02-f888fb741327-image-registry-private-configuration\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.761903 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.761663 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwswx\" (UniqueName: \"kubernetes.io/projected/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-kube-api-access-gwswx\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:00.846577 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.846502 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:00.862641 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.862617 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/379aa0e8-9d7d-42dc-bd0d-acb425c71a2d-cert\") pod \"ingress-canary-dp7jn\" (UID: \"379aa0e8-9d7d-42dc-bd0d-acb425c71a2d\") " pod="openshift-ingress-canary/ingress-canary-dp7jn" Sep 11 15:09:00.862830 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.862656 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:00.862830 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.862696 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20d09616-cdc7-41d0-9a02-f888fb741327-image-registry-private-configuration\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.862830 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.862726 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwswx\" (UniqueName: \"kubernetes.io/projected/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-kube-api-access-gwswx\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:00.862830 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.862794 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-bound-sa-token\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.863002 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.862839 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:09:00.863002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.862861 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20d09616-cdc7-41d0-9a02-f888fb741327-registry-certificates\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.863002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.862887 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20d09616-cdc7-41d0-9a02-f888fb741327-trusted-ca\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.863002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.862913 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20d09616-cdc7-41d0-9a02-f888fb741327-installation-pull-secrets\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.863002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.862939 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-config-volume\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:00.863002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.862966 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwn98\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-kube-api-access-pwn98\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.863002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.862990 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-tmp-dir\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:00.863304 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.863050 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20d09616-cdc7-41d0-9a02-f888fb741327-ca-trust-extracted\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.863304 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.863072 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbgsv\" (UniqueName: \"kubernetes.io/projected/379aa0e8-9d7d-42dc-bd0d-acb425c71a2d-kube-api-access-zbgsv\") pod \"ingress-canary-dp7jn\" (UID: \"379aa0e8-9d7d-42dc-bd0d-acb425c71a2d\") " pod="openshift-ingress-canary/ingress-canary-dp7jn" Sep 11 15:09:00.863304 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.863092 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.863304 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.863183 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:09:00.863304 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.863196 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fddbb4949-qnxrp: secret "image-registry-tls" not found Sep 11 15:09:00.863548 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.863526 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-tmp-dir\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:00.863611 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.863560 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls podName:584cffdd-40bc-4fd9-bd8f-fb92dd86978e nodeName:}" failed. No retries permitted until 2026-09-11 15:09:01.363246695 +0000 UTC m=+35.595171110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls") pod "dns-default-tngb6" (UID: "584cffdd-40bc-4fd9-bd8f-fb92dd86978e") : secret "dns-default-metrics-tls" not found Sep 11 15:09:00.863611 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:00.863584 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls podName:20d09616-cdc7-41d0-9a02-f888fb741327 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:01.363575074 +0000 UTC m=+35.595499487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls") pod "image-registry-7fddbb4949-qnxrp" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327") : secret "image-registry-tls" not found Sep 11 15:09:00.863752 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.863713 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20d09616-cdc7-41d0-9a02-f888fb741327-registry-certificates\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.863866 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.863837 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20d09616-cdc7-41d0-9a02-f888fb741327-ca-trust-extracted\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.863987 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.863863 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-config-volume\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:00.864624 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.864582 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20d09616-cdc7-41d0-9a02-f888fb741327-trusted-ca\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.867392 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.867369 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20d09616-cdc7-41d0-9a02-f888fb741327-image-registry-private-configuration\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.867512 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.867401 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20d09616-cdc7-41d0-9a02-f888fb741327-installation-pull-secrets\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.867512 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.867414 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/379aa0e8-9d7d-42dc-bd0d-acb425c71a2d-cert\") pod \"ingress-canary-dp7jn\" (UID: \"379aa0e8-9d7d-42dc-bd0d-acb425c71a2d\") " pod="openshift-ingress-canary/ingress-canary-dp7jn" Sep 11 15:09:00.871623 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.871580 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwswx\" (UniqueName: \"kubernetes.io/projected/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-kube-api-access-gwswx\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:00.872354 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.872336 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbgsv\" (UniqueName: \"kubernetes.io/projected/379aa0e8-9d7d-42dc-bd0d-acb425c71a2d-kube-api-access-zbgsv\") pod \"ingress-canary-dp7jn\" (UID: \"379aa0e8-9d7d-42dc-bd0d-acb425c71a2d\") " pod="openshift-ingress-canary/ingress-canary-dp7jn" Sep 11 15:09:00.872522 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.872502 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwn98\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-kube-api-access-pwn98\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.872605 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.872538 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-bound-sa-token\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:00.942682 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:00.942648 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dp7jn" Sep 11 15:09:01.031581 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:01.031548 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:01.258022 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:01.257988 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:09:01.258199 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:01.257988 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:09:01.260647 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:01.260620 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 15:09:01.261640 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:01.261220 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 15:09:01.261640 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:01.261269 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4h8rg\"" Sep 11 15:09:01.261640 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:01.261504 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 15:09:01.261640 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:01.261533 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2f6h7\"" Sep 11 15:09:01.261921 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:01.261646 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dp7jn"] Sep 11 15:09:01.266877 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:09:01.266856 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379aa0e8_9d7d_42dc_bd0d_acb425c71a2d.slice/crio-5d7771b222f7d3af60c0839970879f33a40ab3cfbd230ffd4e7b7a6ba21e500f WatchSource:0}: Error finding container 5d7771b222f7d3af60c0839970879f33a40ab3cfbd230ffd4e7b7a6ba21e500f: Status 404 returned error can't find the container with id 5d7771b222f7d3af60c0839970879f33a40ab3cfbd230ffd4e7b7a6ba21e500f Sep 11 15:09:01.367145 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:01.367108 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:01.367145 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:01.367149 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:01.367339 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:01.367266 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:09:01.367339 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:01.367290 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fddbb4949-qnxrp: secret "image-registry-tls" not found Sep 11 15:09:01.367339 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:01.367304 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:09:01.367436 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:01.367356 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls podName:20d09616-cdc7-41d0-9a02-f888fb741327 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:02.367340237 +0000 UTC m=+36.599264667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls") pod "image-registry-7fddbb4949-qnxrp" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327") : secret "image-registry-tls" not found Sep 11 15:09:01.367436 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:01.367371 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls podName:584cffdd-40bc-4fd9-bd8f-fb92dd86978e nodeName:}" failed. No retries permitted until 2026-09-11 15:09:02.367365305 +0000 UTC m=+36.599289745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls") pod "dns-default-tngb6" (UID: "584cffdd-40bc-4fd9-bd8f-fb92dd86978e") : secret "dns-default-metrics-tls" not found Sep 11 15:09:01.424801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:01.424759 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dp7jn" event={"ID":"379aa0e8-9d7d-42dc-bd0d-acb425c71a2d","Type":"ContainerStarted","Data":"5d7771b222f7d3af60c0839970879f33a40ab3cfbd230ffd4e7b7a6ba21e500f"} Sep 11 15:09:02.258328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:02.257936 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:09:02.260578 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:02.260559 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 15:09:02.378388 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:02.378344 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:02.378569 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:02.378407 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:02.378569 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:02.378490 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:09:02.378569 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:02.378514 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fddbb4949-qnxrp: secret "image-registry-tls" not found Sep 11 15:09:02.378569 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:02.378540 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:09:02.378808 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:02.378591 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls podName:20d09616-cdc7-41d0-9a02-f888fb741327 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:04.378569587 +0000 UTC m=+38.610494002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls") pod "image-registry-7fddbb4949-qnxrp" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327") : secret "image-registry-tls" not found Sep 11 15:09:02.378808 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:02.378608 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls podName:584cffdd-40bc-4fd9-bd8f-fb92dd86978e nodeName:}" failed. No retries permitted until 2026-09-11 15:09:04.378599809 +0000 UTC m=+38.610524222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls") pod "dns-default-tngb6" (UID: "584cffdd-40bc-4fd9-bd8f-fb92dd86978e") : secret "dns-default-metrics-tls" not found Sep 11 15:09:04.396312 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:04.396099 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:04.396792 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:04.396330 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:04.396792 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:04.396272 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:09:04.396792 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:04.396412 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fddbb4949-qnxrp: secret "image-registry-tls" not found Sep 11 15:09:04.396792 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:04.396446 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:09:04.396792 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:04.396494 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls podName:20d09616-cdc7-41d0-9a02-f888fb741327 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:08.396470413 +0000 UTC m=+42.628394842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls") pod "image-registry-7fddbb4949-qnxrp" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327") : secret "image-registry-tls" not found Sep 11 15:09:04.396792 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:04.396514 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls podName:584cffdd-40bc-4fd9-bd8f-fb92dd86978e nodeName:}" failed. No retries permitted until 2026-09-11 15:09:08.39650353 +0000 UTC m=+42.628427941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls") pod "dns-default-tngb6" (UID: "584cffdd-40bc-4fd9-bd8f-fb92dd86978e") : secret "dns-default-metrics-tls" not found Sep 11 15:09:05.786030 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:05.786004 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:05.844369 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:05.844340 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:06.441337 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:06.441299 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dp7jn" event={"ID":"379aa0e8-9d7d-42dc-bd0d-acb425c71a2d","Type":"ContainerStarted","Data":"3330b0bd7914c4948302943fb047172b6b6f5c436fe0980432e8a1db03a8d8bc"} Sep 11 15:09:06.443987 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:06.443962 2558 generic.go:356] "Generic (PLEG): container finished" podID="ac2f5397-4638-4694-9857-e5a25f186318" containerID="7097d868183f77c947ea62a197ee538d1eab553aaceb843fce0312c5e906e226" exitCode=0 Sep 11 15:09:06.444111 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:06.444008 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b84v" event={"ID":"ac2f5397-4638-4694-9857-e5a25f186318","Type":"ContainerDied","Data":"7097d868183f77c947ea62a197ee538d1eab553aaceb843fce0312c5e906e226"} Sep 11 15:09:06.460871 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:06.460830 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dp7jn" podStartSLOduration=1.81620239 podStartE2EDuration="6.460815934s" totalImagesPullingTime="4.644613544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:06.459975553 +0000 UTC m=+40.691899988" watchObservedRunningTime="2026-09-11 15:09:06.460815934 +0000 UTC m=+40.692740364" Sep 11 15:09:07.449350 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:07.449316 2558 generic.go:356] "Generic (PLEG): container finished" podID="ac2f5397-4638-4694-9857-e5a25f186318" containerID="b8f37b168bec9ea7c932f22dbac2d14aee048019a4dd6de11dad2331ae899b92" exitCode=0 Sep 11 15:09:07.449714 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:07.449383 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b84v" event={"ID":"ac2f5397-4638-4694-9857-e5a25f186318","Type":"ContainerDied","Data":"b8f37b168bec9ea7c932f22dbac2d14aee048019a4dd6de11dad2331ae899b92"} Sep 11 15:09:08.200666 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:08.200633 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:08.296263 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:08.296228 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:08.364787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:08.364756 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:08.429978 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:08.429955 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:08.430078 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:08.429986 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:08.430121 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:08.430082 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:09:08.430121 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:08.430089 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:09:08.430121 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:08.430103 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fddbb4949-qnxrp: secret "image-registry-tls" not found Sep 11 15:09:08.430231 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:08.430130 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls podName:584cffdd-40bc-4fd9-bd8f-fb92dd86978e nodeName:}" failed. No retries permitted until 2026-09-11 15:09:16.430116368 +0000 UTC m=+50.662040776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls") pod "dns-default-tngb6" (UID: "584cffdd-40bc-4fd9-bd8f-fb92dd86978e") : secret "dns-default-metrics-tls" not found Sep 11 15:09:08.430231 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:08.430142 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls podName:20d09616-cdc7-41d0-9a02-f888fb741327 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:16.43013671 +0000 UTC m=+50.662061119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls") pod "image-registry-7fddbb4949-qnxrp" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327") : secret "image-registry-tls" not found Sep 11 15:09:08.455095 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:08.455020 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b84v" event={"ID":"ac2f5397-4638-4694-9857-e5a25f186318","Type":"ContainerStarted","Data":"07eac950b9501f389c68302a83b779a51a9cb46f19fc1c10b0014c6563666742"} Sep 11 15:09:08.480353 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:08.480308 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6b84v" podStartSLOduration=7.384771825 podStartE2EDuration="41.480293749s" totalImagesPullingTime="34.095521924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:08.478975768 +0000 UTC m=+42.710900199" watchObservedRunningTime="2026-09-11 15:09:08.480293749 +0000 UTC m=+42.712218177" Sep 11 15:09:08.605293 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:08.605264 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:08.663897 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:08.663870 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:09.249279 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:09.249244 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:09.650072 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:09.650044 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:09.783883 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:09.783856 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:10.287511 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:10.287477 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:12.690499 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:12.690474 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:12.745980 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:12.745956 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:12.890164 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:12.890095 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:12.953052 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:12.953019 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:15.409058 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:15.409018 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:16.491208 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:16.491177 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:16.491208 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:16.491210 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:16.491767 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:16.491375 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:09:16.491767 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:16.491404 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fddbb4949-qnxrp: secret "image-registry-tls" not found Sep 11 15:09:16.491767 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:16.491381 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:09:16.491767 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:16.491486 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls podName:20d09616-cdc7-41d0-9a02-f888fb741327 nodeName:}" failed. No retries permitted until 2026-09-11 15:09:32.491463277 +0000 UTC m=+66.723387705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls") pod "image-registry-7fddbb4949-qnxrp" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327") : secret "image-registry-tls" not found Sep 11 15:09:16.491767 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:16.491510 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls podName:584cffdd-40bc-4fd9-bd8f-fb92dd86978e nodeName:}" failed. No retries permitted until 2026-09-11 15:09:32.491498459 +0000 UTC m=+66.723422867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls") pod "dns-default-tngb6" (UID: "584cffdd-40bc-4fd9-bd8f-fb92dd86978e") : secret "dns-default-metrics-tls" not found Sep 11 15:09:21.573930 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.573900 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:21.581762 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.581724 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h"] Sep 11 15:09:21.585997 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.585977 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" Sep 11 15:09:21.588169 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.588147 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 11 15:09:21.588306 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.588208 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 11 15:09:21.588472 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.588453 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:09:21.589136 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.589121 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-t742m\"" Sep 11 15:09:21.597001 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.596976 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h"] Sep 11 15:09:21.625166 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.625146 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscjr\" (UniqueName: \"kubernetes.io/projected/68b94013-2839-4cd9-af6c-b5c8dabc607d-kube-api-access-nscjr\") pod \"cluster-samples-operator-86499c89c4-khl8h\" (UID: \"68b94013-2839-4cd9-af6c-b5c8dabc607d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" Sep 11 15:09:21.625266 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.625214 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-khl8h\" (UID: \"68b94013-2839-4cd9-af6c-b5c8dabc607d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" Sep 11 15:09:21.646725 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.646701 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:21.669645 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.669621 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq"] Sep 11 15:09:21.673395 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.673379 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" Sep 11 15:09:21.675916 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.675859 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:09:21.676125 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.676105 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lkswv\"" Sep 11 15:09:21.676238 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.676189 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 11 15:09:21.676371 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.676352 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 11 15:09:21.676498 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.676482 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 11 15:09:21.683440 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.683420 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq"] Sep 11 15:09:21.726058 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.726030 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8s9n\" (UniqueName: \"kubernetes.io/projected/598ebdf6-3cba-4a77-b9ab-b8edaee3db5f-kube-api-access-j8s9n\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xdjlq\" (UID: \"598ebdf6-3cba-4a77-b9ab-b8edaee3db5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" Sep 11 15:09:21.726161 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.726098 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-khl8h\" (UID: \"68b94013-2839-4cd9-af6c-b5c8dabc607d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" Sep 11 15:09:21.726161 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.726155 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscjr\" (UniqueName: \"kubernetes.io/projected/68b94013-2839-4cd9-af6c-b5c8dabc607d-kube-api-access-nscjr\") pod \"cluster-samples-operator-86499c89c4-khl8h\" (UID: \"68b94013-2839-4cd9-af6c-b5c8dabc607d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" Sep 11 15:09:21.726229 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.726180 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/598ebdf6-3cba-4a77-b9ab-b8edaee3db5f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xdjlq\" (UID: \"598ebdf6-3cba-4a77-b9ab-b8edaee3db5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" Sep 11 15:09:21.726229 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:21.726180 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 15:09:21.726293 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.726234 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/598ebdf6-3cba-4a77-b9ab-b8edaee3db5f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xdjlq\" (UID: \"598ebdf6-3cba-4a77-b9ab-b8edaee3db5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" Sep 11 15:09:21.726349 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:21.726328 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls podName:68b94013-2839-4cd9-af6c-b5c8dabc607d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:22.226314664 +0000 UTC m=+56.458239072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-khl8h" (UID: "68b94013-2839-4cd9-af6c-b5c8dabc607d") : secret "samples-operator-tls" not found Sep 11 15:09:21.735326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.735300 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscjr\" (UniqueName: \"kubernetes.io/projected/68b94013-2839-4cd9-af6c-b5c8dabc607d-kube-api-access-nscjr\") pod \"cluster-samples-operator-86499c89c4-khl8h\" (UID: \"68b94013-2839-4cd9-af6c-b5c8dabc607d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" Sep 11 15:09:21.814294 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.814232 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:21.827353 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.827330 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8s9n\" (UniqueName: \"kubernetes.io/projected/598ebdf6-3cba-4a77-b9ab-b8edaee3db5f-kube-api-access-j8s9n\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xdjlq\" (UID: \"598ebdf6-3cba-4a77-b9ab-b8edaee3db5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" Sep 11 15:09:21.827455 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.827437 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/598ebdf6-3cba-4a77-b9ab-b8edaee3db5f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xdjlq\" (UID: \"598ebdf6-3cba-4a77-b9ab-b8edaee3db5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" Sep 11 15:09:21.827494 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.827461 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/598ebdf6-3cba-4a77-b9ab-b8edaee3db5f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xdjlq\" (UID: \"598ebdf6-3cba-4a77-b9ab-b8edaee3db5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" Sep 11 15:09:21.828095 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.828074 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/598ebdf6-3cba-4a77-b9ab-b8edaee3db5f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xdjlq\" (UID: \"598ebdf6-3cba-4a77-b9ab-b8edaee3db5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" Sep 11 15:09:21.829513 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.829487 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/598ebdf6-3cba-4a77-b9ab-b8edaee3db5f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xdjlq\" (UID: \"598ebdf6-3cba-4a77-b9ab-b8edaee3db5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" Sep 11 15:09:21.834584 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.834559 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8s9n\" (UniqueName: \"kubernetes.io/projected/598ebdf6-3cba-4a77-b9ab-b8edaee3db5f-kube-api-access-j8s9n\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xdjlq\" (UID: \"598ebdf6-3cba-4a77-b9ab-b8edaee3db5f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" Sep 11 15:09:21.872807 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.872784 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:21.981498 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:21.981461 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" Sep 11 15:09:22.230935 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:22.230896 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-khl8h\" (UID: \"68b94013-2839-4cd9-af6c-b5c8dabc607d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" Sep 11 15:09:22.231146 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:22.231077 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 15:09:22.231217 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:22.231158 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls podName:68b94013-2839-4cd9-af6c-b5c8dabc607d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:23.231137607 +0000 UTC m=+57.463062051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-khl8h" (UID: "68b94013-2839-4cd9-af6c-b5c8dabc607d") : secret "samples-operator-tls" not found Sep 11 15:09:22.250433 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:22.250407 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq"] Sep 11 15:09:22.254055 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:09:22.254025 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598ebdf6_3cba_4a77_b9ab_b8edaee3db5f.slice/crio-9b85f770516c1a89e57fef6860fc2aa1ae9411ec58d4c81b5b8e219fceffc3dc WatchSource:0}: Error finding container 9b85f770516c1a89e57fef6860fc2aa1ae9411ec58d4c81b5b8e219fceffc3dc: Status 404 returned error can't find the container with id 9b85f770516c1a89e57fef6860fc2aa1ae9411ec58d4c81b5b8e219fceffc3dc Sep 11 15:09:22.489658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:22.489571 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" event={"ID":"598ebdf6-3cba-4a77-b9ab-b8edaee3db5f","Type":"ContainerStarted","Data":"9b85f770516c1a89e57fef6860fc2aa1ae9411ec58d4c81b5b8e219fceffc3dc"} Sep 11 15:09:23.238444 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:23.238409 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-khl8h\" (UID: \"68b94013-2839-4cd9-af6c-b5c8dabc607d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" Sep 11 15:09:23.238881 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:23.238584 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 15:09:23.238881 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:23.238669 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls podName:68b94013-2839-4cd9-af6c-b5c8dabc607d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:25.23864961 +0000 UTC m=+59.470574042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-khl8h" (UID: "68b94013-2839-4cd9-af6c-b5c8dabc607d") : secret "samples-operator-tls" not found Sep 11 15:09:23.327669 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:23.327628 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:23.381502 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:23.381471 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:24.321833 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:24.321799 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:24.421010 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:24.420979 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:24.494915 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:24.494887 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" event={"ID":"598ebdf6-3cba-4a77-b9ab-b8edaee3db5f","Type":"ContainerStarted","Data":"807049c4b8a1efda11898607d30d4dcc0304973596e23e57c1e07490977c4cdd"} Sep 11 15:09:24.514936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:24.514883 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" podStartSLOduration=1.423094865 podStartE2EDuration="3.514862729s" totalImagesPullingTime="2.091767864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:24.513532478 +0000 UTC m=+58.745456962" watchObservedRunningTime="2026-09-11 15:09:24.514862729 +0000 UTC m=+58.746787162" Sep 11 15:09:24.522096 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:24.522069 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:25.253353 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:25.253319 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-khl8h\" (UID: \"68b94013-2839-4cd9-af6c-b5c8dabc607d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" Sep 11 15:09:25.253532 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:25.253455 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 15:09:25.253532 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:25.253523 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls podName:68b94013-2839-4cd9-af6c-b5c8dabc607d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:29.253506936 +0000 UTC m=+63.485431358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-khl8h" (UID: "68b94013-2839-4cd9-af6c-b5c8dabc607d") : secret "samples-operator-tls" not found Sep 11 15:09:25.988801 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:25.988771 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:26.051960 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.051925 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:26.344912 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.344847 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:26.400032 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.400004 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:26.503513 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.503483 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8xm5j"] Sep 11 15:09:26.537696 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.537669 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:26.540329 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.540304 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 15:09:26.540442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.540305 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 15:09:26.540442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.540317 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 15:09:26.540921 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.540908 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cr68x\"" Sep 11 15:09:26.541094 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.541082 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 15:09:26.545414 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.545394 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8xm5j"] Sep 11 15:09:26.547038 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.547013 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:26.562293 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.562271 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69574f6d-1b8f-4d63-b24d-119756a0267d-data-volume\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:26.562400 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.562297 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x6zd\" (UniqueName: \"kubernetes.io/projected/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-api-access-6x6zd\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:26.562400 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.562316 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:26.562400 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.562346 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69574f6d-1b8f-4d63-b24d-119756a0267d-crio-socket\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:26.562562 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.562505 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:26.636834 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.636783 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:26.663037 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.663014 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69574f6d-1b8f-4d63-b24d-119756a0267d-data-volume\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:26.663321 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.663044 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x6zd\" (UniqueName: \"kubernetes.io/projected/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-api-access-6x6zd\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:26.663321 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.663064 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:26.663321 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:26.663166 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:09:26.663321 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:26.663213 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls podName:69574f6d-1b8f-4d63-b24d-119756a0267d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:27.163198888 +0000 UTC m=+61.395123309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xm5j" (UID: "69574f6d-1b8f-4d63-b24d-119756a0267d") : secret "insights-runtime-extractor-tls" not found Sep 11 15:09:26.663486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.663371 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69574f6d-1b8f-4d63-b24d-119756a0267d-crio-socket\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:26.663486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.663437 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69574f6d-1b8f-4d63-b24d-119756a0267d-crio-socket\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:26.663486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.663466 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:26.663583 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:26.663513 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:09:26.663583 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:26.663573 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm podName:69574f6d-1b8f-4d63-b24d-119756a0267d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:27.16355725 +0000 UTC m=+61.395481673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xm5j" (UID: "69574f6d-1b8f-4d63-b24d-119756a0267d") : configmap "kube-rbac-proxy" not found Sep 11 15:09:26.663583 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.663574 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69574f6d-1b8f-4d63-b24d-119756a0267d-data-volume\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:26.671337 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:26.671316 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x6zd\" (UniqueName: \"kubernetes.io/projected/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-api-access-6x6zd\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:27.139427 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:27.139389 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:27.166753 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:27.166719 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:27.166898 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:27.166829 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:09:27.166898 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:27.166865 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:27.166997 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:27.166924 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm podName:69574f6d-1b8f-4d63-b24d-119756a0267d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:28.166902386 +0000 UTC m=+62.398826812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xm5j" (UID: "69574f6d-1b8f-4d63-b24d-119756a0267d") : configmap "kube-rbac-proxy" not found Sep 11 15:09:27.166997 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:27.166975 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:09:27.167108 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:27.167017 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls podName:69574f6d-1b8f-4d63-b24d-119756a0267d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:28.167006938 +0000 UTC m=+62.398931346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xm5j" (UID: "69574f6d-1b8f-4d63-b24d-119756a0267d") : secret "insights-runtime-extractor-tls" not found Sep 11 15:09:27.203963 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:27.203936 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:27.421179 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:27.421154 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbxnk" Sep 11 15:09:27.581175 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:27.581144 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:27.678701 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:27.678636 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:28.175749 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.175716 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:28.176107 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.175811 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:28.176107 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:28.175843 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:09:28.176107 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:28.175912 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm podName:69574f6d-1b8f-4d63-b24d-119756a0267d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:30.175891812 +0000 UTC m=+64.407816248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xm5j" (UID: "69574f6d-1b8f-4d63-b24d-119756a0267d") : configmap "kube-rbac-proxy" not found Sep 11 15:09:28.176107 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:28.175923 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:09:28.176107 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:28.175957 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls podName:69574f6d-1b8f-4d63-b24d-119756a0267d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:30.175947466 +0000 UTC m=+64.407871879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xm5j" (UID: "69574f6d-1b8f-4d63-b24d-119756a0267d") : secret "insights-runtime-extractor-tls" not found Sep 11 15:09:28.653909 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.653875 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-n9bjv"] Sep 11 15:09:28.656735 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.656714 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" Sep 11 15:09:28.659609 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.659592 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 11 15:09:28.660304 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.660281 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 11 15:09:28.660393 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.660310 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-npxgl\"" Sep 11 15:09:28.660393 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.660339 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 11 15:09:28.660484 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.660444 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 11 15:09:28.665543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.665525 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-n9bjv"] Sep 11 15:09:28.679443 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.679421 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06f3adad-f49b-437c-91cb-5f7cf6f460c1-signing-key\") pod \"service-ca-568869fdc5-n9bjv\" (UID: \"06f3adad-f49b-437c-91cb-5f7cf6f460c1\") " pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" Sep 11 15:09:28.679528 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.679448 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06f3adad-f49b-437c-91cb-5f7cf6f460c1-signing-cabundle\") pod \"service-ca-568869fdc5-n9bjv\" (UID: \"06f3adad-f49b-437c-91cb-5f7cf6f460c1\") " pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" Sep 11 15:09:28.679528 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.679479 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmkkg\" (UniqueName: \"kubernetes.io/projected/06f3adad-f49b-437c-91cb-5f7cf6f460c1-kube-api-access-kmkkg\") pod \"service-ca-568869fdc5-n9bjv\" (UID: \"06f3adad-f49b-437c-91cb-5f7cf6f460c1\") " pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" Sep 11 15:09:28.763571 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.763544 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:28.780808 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.780787 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmkkg\" (UniqueName: \"kubernetes.io/projected/06f3adad-f49b-437c-91cb-5f7cf6f460c1-kube-api-access-kmkkg\") pod \"service-ca-568869fdc5-n9bjv\" (UID: \"06f3adad-f49b-437c-91cb-5f7cf6f460c1\") " pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" Sep 11 15:09:28.780918 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.780903 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06f3adad-f49b-437c-91cb-5f7cf6f460c1-signing-key\") pod \"service-ca-568869fdc5-n9bjv\" (UID: \"06f3adad-f49b-437c-91cb-5f7cf6f460c1\") " pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" Sep 11 15:09:28.780990 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.780924 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06f3adad-f49b-437c-91cb-5f7cf6f460c1-signing-cabundle\") pod \"service-ca-568869fdc5-n9bjv\" (UID: \"06f3adad-f49b-437c-91cb-5f7cf6f460c1\") " pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" Sep 11 15:09:28.781526 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.781505 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06f3adad-f49b-437c-91cb-5f7cf6f460c1-signing-cabundle\") pod \"service-ca-568869fdc5-n9bjv\" (UID: \"06f3adad-f49b-437c-91cb-5f7cf6f460c1\") " pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" Sep 11 15:09:28.783234 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.783214 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06f3adad-f49b-437c-91cb-5f7cf6f460c1-signing-key\") pod \"service-ca-568869fdc5-n9bjv\" (UID: \"06f3adad-f49b-437c-91cb-5f7cf6f460c1\") " pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" Sep 11 15:09:28.788898 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.788880 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmkkg\" (UniqueName: \"kubernetes.io/projected/06f3adad-f49b-437c-91cb-5f7cf6f460c1-kube-api-access-kmkkg\") pod \"service-ca-568869fdc5-n9bjv\" (UID: \"06f3adad-f49b-437c-91cb-5f7cf6f460c1\") " pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" Sep 11 15:09:28.864287 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.864258 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:28.966238 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:28.966175 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" Sep 11 15:09:29.241303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:29.241219 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-n9bjv"] Sep 11 15:09:29.244255 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:09:29.244227 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f3adad_f49b_437c_91cb_5f7cf6f460c1.slice/crio-c064153bbd41205087236cc6490acab5aba9d1e4cbc574bb81f0afadb3aa19fe WatchSource:0}: Error finding container c064153bbd41205087236cc6490acab5aba9d1e4cbc574bb81f0afadb3aa19fe: Status 404 returned error can't find the container with id c064153bbd41205087236cc6490acab5aba9d1e4cbc574bb81f0afadb3aa19fe Sep 11 15:09:29.285149 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:29.285124 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-khl8h\" (UID: \"68b94013-2839-4cd9-af6c-b5c8dabc607d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" Sep 11 15:09:29.285304 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:29.285285 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 15:09:29.285382 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:29.285369 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls podName:68b94013-2839-4cd9-af6c-b5c8dabc607d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:37.285347954 +0000 UTC m=+71.517272366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-khl8h" (UID: "68b94013-2839-4cd9-af6c-b5c8dabc607d") : secret "samples-operator-tls" not found Sep 11 15:09:29.293804 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:29.293782 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r85xv_d209e966-9242-4f9d-8d2d-63dc61ffc5c3/dns-node-resolver/0.log" Sep 11 15:09:29.508323 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:29.508237 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" event={"ID":"06f3adad-f49b-437c-91cb-5f7cf6f460c1","Type":"ContainerStarted","Data":"c064153bbd41205087236cc6490acab5aba9d1e4cbc574bb81f0afadb3aa19fe"} Sep 11 15:09:29.745344 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:29.745302 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:30.091401 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:30.091373 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4mzss_6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb/node-ca/0.log" Sep 11 15:09:30.192533 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:30.192502 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:30.192702 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:30.192574 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:30.192702 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:30.192626 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:09:30.192702 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:30.192696 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm podName:69574f6d-1b8f-4d63-b24d-119756a0267d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:34.192677619 +0000 UTC m=+68.424602051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xm5j" (UID: "69574f6d-1b8f-4d63-b24d-119756a0267d") : configmap "kube-rbac-proxy" not found Sep 11 15:09:30.192903 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:30.192767 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:09:30.192903 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:30.192833 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls podName:69574f6d-1b8f-4d63-b24d-119756a0267d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:34.192814573 +0000 UTC m=+68.424738993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xm5j" (UID: "69574f6d-1b8f-4d63-b24d-119756a0267d") : secret "insights-runtime-extractor-tls" not found Sep 11 15:09:30.890384 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:30.890351 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dp7jn_379aa0e8-9d7d-42dc-bd0d-acb425c71a2d/serve-healthcheck-canary/0.log" Sep 11 15:09:31.515001 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:31.514928 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" event={"ID":"06f3adad-f49b-437c-91cb-5f7cf6f460c1","Type":"ContainerStarted","Data":"82bb669251ebe5eafbe02d18cde26dcef31efd50b57b1d1577ccdefbd0b3c011"} Sep 11 15:09:31.531708 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:31.531657 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-n9bjv" podStartSLOduration=1.62018597 podStartE2EDuration="3.531636317s" totalImagesPullingTime="1.911450347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:31.53086154 +0000 UTC m=+65.762785982" watchObservedRunningTime="2026-09-11 15:09:31.531636317 +0000 UTC m=+65.763560749" Sep 11 15:09:31.893117 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:31.893091 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-xdjlq_598ebdf6-3cba-4a77-b9ab-b8edaee3db5f/kube-storage-version-migrator-operator/0.log" Sep 11 15:09:32.109133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.109093 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:09:32.109309 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.109152 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:09:32.111872 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.111846 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 15:09:32.111998 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.111878 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 15:09:32.120158 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:32.120136 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 15:09:32.120273 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:32.120205 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs podName:aaf8fe6d-2de1-4a11-9798-5d3e2e83164e nodeName:}" failed. No retries permitted until 2026-09-11 15:10:36.120183013 +0000 UTC m=+130.352107422 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs") pod "network-metrics-daemon-lkx5k" (UID: "aaf8fe6d-2de1-4a11-9798-5d3e2e83164e") : secret "metrics-daemon-secret" not found Sep 11 15:09:32.123439 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.123413 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f547eaf0-def0-4524-9755-e682cd106e33-original-pull-secret\") pod \"global-pull-secret-syncer-cdnlq\" (UID: \"f547eaf0-def0-4524-9755-e682cd106e33\") " pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:09:32.210087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.209993 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9q8\" (UniqueName: \"kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8\") pod \"network-check-target-qcg9p\" (UID: \"1037cb0e-feaa-44fa-a256-9e9143945544\") " pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:09:32.212468 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.212429 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 15:09:32.223039 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.222840 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 15:09:32.234219 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.234188 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9q8\" (UniqueName: \"kubernetes.io/projected/1037cb0e-feaa-44fa-a256-9e9143945544-kube-api-access-bz9q8\") pod \"network-check-target-qcg9p\" (UID: \"1037cb0e-feaa-44fa-a256-9e9143945544\") " pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:09:32.256843 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.256809 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:32.268833 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.268810 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cdnlq" Sep 11 15:09:32.479387 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.479297 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2f6h7\"" Sep 11 15:09:32.487534 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.487499 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:09:32.513400 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.513063 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:09:32.513400 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.513121 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:09:32.513400 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:32.513246 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:09:32.513400 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:32.513277 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:09:32.513400 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:32.513302 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fddbb4949-qnxrp: secret "image-registry-tls" not found Sep 11 15:09:32.513400 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:32.513305 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls podName:584cffdd-40bc-4fd9-bd8f-fb92dd86978e nodeName:}" failed. No retries permitted until 2026-09-11 15:10:04.513286874 +0000 UTC m=+98.745211291 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls") pod "dns-default-tngb6" (UID: "584cffdd-40bc-4fd9-bd8f-fb92dd86978e") : secret "dns-default-metrics-tls" not found Sep 11 15:09:32.513400 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:32.513367 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls podName:20d09616-cdc7-41d0-9a02-f888fb741327 nodeName:}" failed. No retries permitted until 2026-09-11 15:10:04.513349525 +0000 UTC m=+98.745273941 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls") pod "image-registry-7fddbb4949-qnxrp" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327") : secret "image-registry-tls" not found Sep 11 15:09:32.520328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.520294 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:32.570939 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.570866 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cdnlq"] Sep 11 15:09:32.574818 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:09:32.574779 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf547eaf0_def0_4524_9755_e682cd106e33.slice/crio-ac81de8a7319cff3ea82d65c50676930d1d5379db3accc0dde69a1eaab768fea WatchSource:0}: Error finding container ac81de8a7319cff3ea82d65c50676930d1d5379db3accc0dde69a1eaab768fea: Status 404 returned error can't find the container with id ac81de8a7319cff3ea82d65c50676930d1d5379db3accc0dde69a1eaab768fea Sep 11 15:09:32.623467 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.623386 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:32.720593 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.720565 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:32.776958 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.776886 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qcg9p"] Sep 11 15:09:32.780503 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:09:32.780479 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1037cb0e_feaa_44fa_a256_9e9143945544.slice/crio-35f690457e41e9481b79eaf57ad63ffb5398514a07c16503ea5d1f0b5d0b9693 WatchSource:0}: Error finding container 35f690457e41e9481b79eaf57ad63ffb5398514a07c16503ea5d1f0b5d0b9693: Status 404 returned error can't find the container with id 35f690457e41e9481b79eaf57ad63ffb5398514a07c16503ea5d1f0b5d0b9693 Sep 11 15:09:32.947234 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:32.947203 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:33.293388 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:33.293350 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:33.521631 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:33.521593 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qcg9p" event={"ID":"1037cb0e-feaa-44fa-a256-9e9143945544","Type":"ContainerStarted","Data":"35f690457e41e9481b79eaf57ad63ffb5398514a07c16503ea5d1f0b5d0b9693"} Sep 11 15:09:33.522763 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:33.522724 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cdnlq" event={"ID":"f547eaf0-def0-4524-9755-e682cd106e33","Type":"ContainerStarted","Data":"ac81de8a7319cff3ea82d65c50676930d1d5379db3accc0dde69a1eaab768fea"} Sep 11 15:09:33.996521 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:33.996486 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:34.228089 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:34.228044 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:34.228257 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:34.228175 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:34.228257 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:34.228211 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:09:34.228390 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:34.228280 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:09:34.228390 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:34.228305 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls podName:69574f6d-1b8f-4d63-b24d-119756a0267d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:42.228281238 +0000 UTC m=+76.460205653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xm5j" (UID: "69574f6d-1b8f-4d63-b24d-119756a0267d") : secret "insights-runtime-extractor-tls" not found Sep 11 15:09:34.228390 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:34.228342 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm podName:69574f6d-1b8f-4d63-b24d-119756a0267d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:42.228325629 +0000 UTC m=+76.460250055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8xm5j" (UID: "69574f6d-1b8f-4d63-b24d-119756a0267d") : configmap "kube-rbac-proxy" not found Sep 11 15:09:34.345465 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:34.345369 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:34.546363 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:34.546296 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:34.763277 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:34.763243 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:34.953757 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:34.953537 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:35.730128 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:35.729872 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:35.810072 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:35.810034 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:35.936888 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:35.936853 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:35.995851 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:35.995734 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:36.346973 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:36.346895 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:36.398817 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:36.398784 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:36.796898 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:36.796827 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:37.345174 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:37.345139 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:37.356479 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:37.356448 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-khl8h\" (UID: \"68b94013-2839-4cd9-af6c-b5c8dabc607d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" Sep 11 15:09:37.359217 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:37.359191 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/68b94013-2839-4cd9-af6c-b5c8dabc607d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-khl8h\" (UID: \"68b94013-2839-4cd9-af6c-b5c8dabc607d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" Sep 11 15:09:37.496933 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:37.496898 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-t742m\"" Sep 11 15:09:37.505579 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:37.505552 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" Sep 11 15:09:37.542058 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:37.541997 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:37.941633 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:37.941601 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:38.070111 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:38.070035 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h"] Sep 11 15:09:38.451754 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:38.451714 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:38.538480 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:38.538442 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qcg9p" event={"ID":"1037cb0e-feaa-44fa-a256-9e9143945544","Type":"ContainerStarted","Data":"c9567141d50891fa030264ab50a1c0f36253c4760003214c6e65e89a8b20f32e"} Sep 11 15:09:38.538653 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:38.538531 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:09:38.540060 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:38.540034 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cdnlq" event={"ID":"f547eaf0-def0-4524-9755-e682cd106e33","Type":"ContainerStarted","Data":"a889dc55f6916a4df032adea93459a70714ed751f97daeb1e0e2e19e62ccbd94"} Sep 11 15:09:38.541247 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:38.541224 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" event={"ID":"68b94013-2839-4cd9-af6c-b5c8dabc607d","Type":"ContainerStarted","Data":"9c2f01d6355b53c965b69c71cbee39def1b2be97dc189ee81b72bbfabee890c7"} Sep 11 15:09:38.553639 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:38.553604 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qcg9p" podStartSLOduration=66.532664989 podStartE2EDuration="1m11.553591698s" totalImagesPullingTime="5.020926709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:38.552536811 +0000 UTC m=+72.784461241" watchObservedRunningTime="2026-09-11 15:09:38.553591698 +0000 UTC m=+72.785516142" Sep 11 15:09:38.554768 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:38.554719 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:39.089712 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:39.089676 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:39.190907 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:39.190875 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:39.354152 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:39.354073 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:39.744599 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:39.744550 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:39.968638 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:39.968566 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:40.345574 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:40.345542 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:40.912731 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:40.912703 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:41.008753 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:41.008714 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:41.362565 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:41.362492 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:41.543518 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:41.543490 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:41.552915 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:41.552886 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" event={"ID":"68b94013-2839-4cd9-af6c-b5c8dabc607d","Type":"ContainerStarted","Data":"710394973dc8d9518de3052dcb842d7cecb89015be9e5c6dd40200d30b309596"} Sep 11 15:09:41.553052 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:41.552931 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" event={"ID":"68b94013-2839-4cd9-af6c-b5c8dabc607d","Type":"ContainerStarted","Data":"376385e35e2467c8620dc1ad96026ffe9c98a205451f5d281bdc91e92fd1e5e6"} Sep 11 15:09:41.570221 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:41.570177 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-khl8h" podStartSLOduration=18.165799057 podStartE2EDuration="20.570161978s" totalImagesPullingTime="2.404362921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:41.569244537 +0000 UTC m=+75.801168968" watchObservedRunningTime="2026-09-11 15:09:41.570161978 +0000 UTC m=+75.802086409" Sep 11 15:09:41.570355 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:41.570250 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cdnlq" podStartSLOduration=69.332919787 podStartE2EDuration="1m14.570246361s" totalImagesPullingTime="5.237326574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:38.565780529 +0000 UTC m=+72.797704963" watchObservedRunningTime="2026-09-11 15:09:41.570246361 +0000 UTC m=+75.802170791" Sep 11 15:09:41.953716 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:41.953687 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:42.301933 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:42.301858 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:42.301933 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:42.301921 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:42.302129 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:42.301996 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:09:42.302129 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:42.302063 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls podName:69574f6d-1b8f-4d63-b24d-119756a0267d nodeName:}" failed. No retries permitted until 2026-09-11 15:09:58.302049243 +0000 UTC m=+92.533973664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8xm5j" (UID: "69574f6d-1b8f-4d63-b24d-119756a0267d") : secret "insights-runtime-extractor-tls" not found Sep 11 15:09:42.302364 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:42.302348 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69574f6d-1b8f-4d63-b24d-119756a0267d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:42.561513 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:42.561437 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:42.731488 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:42.731456 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:42.820456 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:42.820385 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:42.958917 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:42.958890 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:48.751975 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.751830 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm"] Sep 11 15:09:48.756255 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.756232 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm" Sep 11 15:09:48.759226 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.759202 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 11 15:09:48.759345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.759299 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 11 15:09:48.759345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.759301 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7txf9\"" Sep 11 15:09:48.764631 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.764612 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw"] Sep 11 15:09:48.767428 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.767411 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw" Sep 11 15:09:48.769779 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.769758 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 15:09:48.769883 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.769769 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 15:09:48.770129 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.770107 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm"] Sep 11 15:09:48.770213 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.770140 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 15:09:48.770213 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.770106 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 11 15:09:48.770213 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.770183 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pmhw4\"" Sep 11 15:09:48.782364 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.782342 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw"] Sep 11 15:09:48.847588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.847559 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1a738b7-2dd6-4672-a3d1-866ff1a01b15-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zk9nm\" (UID: \"c1a738b7-2dd6-4672-a3d1-866ff1a01b15\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm" Sep 11 15:09:48.847731 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.847606 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8325e71f-659f-4c71-a0ac-ff84143484c7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-866f76fbd-pt5sw\" (UID: \"8325e71f-659f-4c71-a0ac-ff84143484c7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw" Sep 11 15:09:48.847731 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.847624 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1a738b7-2dd6-4672-a3d1-866ff1a01b15-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-zk9nm\" (UID: \"c1a738b7-2dd6-4672-a3d1-866ff1a01b15\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm" Sep 11 15:09:48.847731 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.847639 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8glz\" (UniqueName: \"kubernetes.io/projected/8325e71f-659f-4c71-a0ac-ff84143484c7-kube-api-access-t8glz\") pod \"managed-serviceaccount-addon-agent-866f76fbd-pt5sw\" (UID: \"8325e71f-659f-4c71-a0ac-ff84143484c7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw" Sep 11 15:09:48.895353 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.895325 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:48.948815 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.948788 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1a738b7-2dd6-4672-a3d1-866ff1a01b15-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-zk9nm\" (UID: \"c1a738b7-2dd6-4672-a3d1-866ff1a01b15\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm" Sep 11 15:09:48.948931 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.948818 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8glz\" (UniqueName: \"kubernetes.io/projected/8325e71f-659f-4c71-a0ac-ff84143484c7-kube-api-access-t8glz\") pod \"managed-serviceaccount-addon-agent-866f76fbd-pt5sw\" (UID: \"8325e71f-659f-4c71-a0ac-ff84143484c7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw" Sep 11 15:09:48.948931 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.948917 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1a738b7-2dd6-4672-a3d1-866ff1a01b15-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zk9nm\" (UID: \"c1a738b7-2dd6-4672-a3d1-866ff1a01b15\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm" Sep 11 15:09:48.949013 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.948972 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8325e71f-659f-4c71-a0ac-ff84143484c7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-866f76fbd-pt5sw\" (UID: \"8325e71f-659f-4c71-a0ac-ff84143484c7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw" Sep 11 15:09:48.949563 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.949544 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1a738b7-2dd6-4672-a3d1-866ff1a01b15-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-zk9nm\" (UID: \"c1a738b7-2dd6-4672-a3d1-866ff1a01b15\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm" Sep 11 15:09:48.951229 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.951214 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c1a738b7-2dd6-4672-a3d1-866ff1a01b15-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zk9nm\" (UID: \"c1a738b7-2dd6-4672-a3d1-866ff1a01b15\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm" Sep 11 15:09:48.951464 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.951445 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8325e71f-659f-4c71-a0ac-ff84143484c7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-866f76fbd-pt5sw\" (UID: \"8325e71f-659f-4c71-a0ac-ff84143484c7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw" Sep 11 15:09:48.958325 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.958302 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8glz\" (UniqueName: \"kubernetes.io/projected/8325e71f-659f-4c71-a0ac-ff84143484c7-kube-api-access-t8glz\") pod \"managed-serviceaccount-addon-agent-866f76fbd-pt5sw\" (UID: \"8325e71f-659f-4c71-a0ac-ff84143484c7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw" Sep 11 15:09:48.997202 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:48.997179 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:49.064931 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:49.064854 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm" Sep 11 15:09:49.076906 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:49.076883 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:49.087473 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:49.087446 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw" Sep 11 15:09:49.148593 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:49.148545 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:49.332806 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:49.332722 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm"] Sep 11 15:09:49.335916 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:09:49.335890 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a738b7_2dd6_4672_a3d1_866ff1a01b15.slice/crio-c8295fcf70748fde46df21c5209830a88a07e925eb86d724408369698a3493d3 WatchSource:0}: Error finding container c8295fcf70748fde46df21c5209830a88a07e925eb86d724408369698a3493d3: Status 404 returned error can't find the container with id c8295fcf70748fde46df21c5209830a88a07e925eb86d724408369698a3493d3 Sep 11 15:09:49.361269 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:49.361247 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw"] Sep 11 15:09:49.363963 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:09:49.363941 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8325e71f_659f_4c71_a0ac_ff84143484c7.slice/crio-9ee2afb32f264dcf28dc46cb228a0d7b02f4188d8e0fae979c0d44815604133d WatchSource:0}: Error finding container 9ee2afb32f264dcf28dc46cb228a0d7b02f4188d8e0fae979c0d44815604133d: Status 404 returned error can't find the container with id 9ee2afb32f264dcf28dc46cb228a0d7b02f4188d8e0fae979c0d44815604133d Sep 11 15:09:49.581154 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:49.581113 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw" event={"ID":"8325e71f-659f-4c71-a0ac-ff84143484c7","Type":"ContainerStarted","Data":"9ee2afb32f264dcf28dc46cb228a0d7b02f4188d8e0fae979c0d44815604133d"} Sep 11 15:09:49.582308 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:49.582275 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm" event={"ID":"c1a738b7-2dd6-4672-a3d1-866ff1a01b15","Type":"ContainerStarted","Data":"c8295fcf70748fde46df21c5209830a88a07e925eb86d724408369698a3493d3"} Sep 11 15:09:50.117874 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.117823 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-52rkt"] Sep 11 15:09:50.121764 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.121197 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-52rkt" Sep 11 15:09:50.122150 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.122051 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:50.128689 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.128655 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 15:09:50.128855 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.128690 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 15:09:50.129602 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.129581 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-llw9q\"" Sep 11 15:09:50.155064 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.155034 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-52rkt"] Sep 11 15:09:50.236473 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.236429 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:50.259322 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.259268 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk95b\" (UniqueName: \"kubernetes.io/projected/9fa10168-da0c-4e3f-aaf6-73db67dd6227-kube-api-access-hk95b\") pod \"downloads-76f99584f4-52rkt\" (UID: \"9fa10168-da0c-4e3f-aaf6-73db67dd6227\") " pod="openshift-console/downloads-76f99584f4-52rkt" Sep 11 15:09:50.360351 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.360004 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk95b\" (UniqueName: \"kubernetes.io/projected/9fa10168-da0c-4e3f-aaf6-73db67dd6227-kube-api-access-hk95b\") pod \"downloads-76f99584f4-52rkt\" (UID: \"9fa10168-da0c-4e3f-aaf6-73db67dd6227\") " pod="openshift-console/downloads-76f99584f4-52rkt" Sep 11 15:09:50.369650 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.369566 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk95b\" (UniqueName: \"kubernetes.io/projected/9fa10168-da0c-4e3f-aaf6-73db67dd6227-kube-api-access-hk95b\") pod \"downloads-76f99584f4-52rkt\" (UID: \"9fa10168-da0c-4e3f-aaf6-73db67dd6227\") " pod="openshift-console/downloads-76f99584f4-52rkt" Sep 11 15:09:50.435193 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.435158 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-52rkt" Sep 11 15:09:50.595236 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.595198 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:50.763666 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.763636 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:50.839667 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:50.839594 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:51.625843 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:51.625813 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:51.725932 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:51.725905 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:51.826716 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:51.826691 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:51.993787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:51.993731 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:52.002220 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:52.002192 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-52rkt"] Sep 11 15:09:52.005777 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:09:52.005754 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa10168_da0c_4e3f_aaf6_73db67dd6227.slice/crio-0db8f749f3c72ca50d917bd99569dedc74464f7590ebb7cf7f447528d76e2267 WatchSource:0}: Error finding container 0db8f749f3c72ca50d917bd99569dedc74464f7590ebb7cf7f447528d76e2267: Status 404 returned error can't find the container with id 0db8f749f3c72ca50d917bd99569dedc74464f7590ebb7cf7f447528d76e2267 Sep 11 15:09:52.185926 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:52.185898 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:52.598639 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:52.598601 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-52rkt" event={"ID":"9fa10168-da0c-4e3f-aaf6-73db67dd6227","Type":"ContainerStarted","Data":"0db8f749f3c72ca50d917bd99569dedc74464f7590ebb7cf7f447528d76e2267"} Sep 11 15:09:52.600129 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:52.600103 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw" event={"ID":"8325e71f-659f-4c71-a0ac-ff84143484c7","Type":"ContainerStarted","Data":"c77aa24e6cdc8f107ce12a434ff91c8f5d93c2112dbe3c1244232443921d49bd"} Sep 11 15:09:52.601692 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:52.601669 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm" event={"ID":"c1a738b7-2dd6-4672-a3d1-866ff1a01b15","Type":"ContainerStarted","Data":"b3114cf47edd060187641142501e31961a932650d503144d1471d301e1b86a76"} Sep 11 15:09:52.634111 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:52.634042 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-866f76fbd-pt5sw" podStartSLOduration=2.20661831 podStartE2EDuration="4.634025245s" totalImagesPullingTime="2.427406935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:52.616177728 +0000 UTC m=+86.848102160" watchObservedRunningTime="2026-09-11 15:09:52.634025245 +0000 UTC m=+86.865949686" Sep 11 15:09:52.709552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:52.709522 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:52.810394 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:52.810363 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:52.911696 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:52.911665 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:54.466533 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:54.466502 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:54.572102 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:54.572072 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:56.215461 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.215370 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:56.350287 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.350249 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:56.355699 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.355647 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zk9nm" podStartSLOduration=5.953361494 podStartE2EDuration="8.355627896s" totalImagesPullingTime="2.402266402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:52.6339828 +0000 UTC m=+86.865907233" watchObservedRunningTime="2026-09-11 15:09:56.355627896 +0000 UTC m=+90.587552328" Sep 11 15:09:56.356688 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.356666 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-28bqs"] Sep 11 15:09:56.359883 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.359866 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-28bqs" Sep 11 15:09:56.362346 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.362325 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-989v8\"" Sep 11 15:09:56.362558 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.362526 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 11 15:09:56.373624 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.373602 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-28bqs"] Sep 11 15:09:56.511504 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.511411 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e41e189f-4474-45ff-acc7-5c7cadf587ed-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-28bqs\" (UID: \"e41e189f-4474-45ff-acc7-5c7cadf587ed\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-28bqs" Sep 11 15:09:56.576526 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.576488 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:56.612533 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.612502 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e41e189f-4474-45ff-acc7-5c7cadf587ed-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-28bqs\" (UID: \"e41e189f-4474-45ff-acc7-5c7cadf587ed\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-28bqs" Sep 11 15:09:56.615534 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.615501 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e41e189f-4474-45ff-acc7-5c7cadf587ed-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-28bqs\" (UID: \"e41e189f-4474-45ff-acc7-5c7cadf587ed\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-28bqs" Sep 11 15:09:56.671896 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.671862 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-28bqs" Sep 11 15:09:56.947311 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:56.947272 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-28bqs"] Sep 11 15:09:56.951050 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:09:56.951026 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41e189f_4474_45ff_acc7_5c7cadf587ed.slice/crio-32aa54071b30be67e4f3232ee79d2114b2bb937f2b43c4d30321f92a40b14ae0 WatchSource:0}: Error finding container 32aa54071b30be67e4f3232ee79d2114b2bb937f2b43c4d30321f92a40b14ae0: Status 404 returned error can't find the container with id 32aa54071b30be67e4f3232ee79d2114b2bb937f2b43c4d30321f92a40b14ae0 Sep 11 15:09:57.604246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:57.604210 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:57.620309 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:57.620274 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-28bqs" event={"ID":"e41e189f-4474-45ff-acc7-5c7cadf587ed","Type":"ContainerStarted","Data":"32aa54071b30be67e4f3232ee79d2114b2bb937f2b43c4d30321f92a40b14ae0"} Sep 11 15:09:58.148601 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.148566 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-865d6dbf9f-6nhcq"] Sep 11 15:09:58.152665 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.152640 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.158132 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.158110 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 15:09:58.158394 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.158372 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 15:09:58.159120 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.159099 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wnkdc\"" Sep 11 15:09:58.159221 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.159129 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 15:09:58.159221 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.159193 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 15:09:58.159328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.159307 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 15:09:58.169346 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.169322 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865d6dbf9f-6nhcq"] Sep 11 15:09:58.326021 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.325991 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-oauth-serving-cert\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.326127 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.326035 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:58.326127 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.326066 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-config\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.326127 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.326105 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-serving-cert\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.326127 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.326122 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b94mt\" (UniqueName: \"kubernetes.io/projected/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-kube-api-access-b94mt\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.326323 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.326149 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-service-ca\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.326323 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.326181 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-oauth-config\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.328682 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.328660 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69574f6d-1b8f-4d63-b24d-119756a0267d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8xm5j\" (UID: \"69574f6d-1b8f-4d63-b24d-119756a0267d\") " pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:58.346761 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.346729 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8xm5j" Sep 11 15:09:58.426834 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.426800 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-serving-cert\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.426967 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.426845 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b94mt\" (UniqueName: \"kubernetes.io/projected/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-kube-api-access-b94mt\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.426967 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.426876 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-service-ca\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.426967 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.426925 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-oauth-config\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.427116 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.427002 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-oauth-serving-cert\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.427116 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.427036 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-config\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.427924 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.427899 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-config\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.427924 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.427919 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-oauth-serving-cert\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.428199 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.428131 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-service-ca\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.429834 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.429805 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-oauth-config\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.431277 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.431256 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-serving-cert\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.438534 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.438510 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b94mt\" (UniqueName: \"kubernetes.io/projected/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-kube-api-access-b94mt\") pod \"console-865d6dbf9f-6nhcq\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.463063 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.463029 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:09:58.625598 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.625556 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-28bqs" event={"ID":"e41e189f-4474-45ff-acc7-5c7cadf587ed","Type":"ContainerStarted","Data":"9d2fc31e15750bf9c264ff33cffaeff8a1caea804a5f32eb08ef643a51e8dcbd"} Sep 11 15:09:58.626104 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.625980 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-28bqs" Sep 11 15:09:58.627654 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.627586 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8xm5j"] Sep 11 15:09:58.630953 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:09:58.630925 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69574f6d_1b8f_4d63_b24d_119756a0267d.slice/crio-935d7a53edf451779af91b035a248bcd34311eec16bf6a2f34d973b22877bd8c WatchSource:0}: Error finding container 935d7a53edf451779af91b035a248bcd34311eec16bf6a2f34d973b22877bd8c: Status 404 returned error can't find the container with id 935d7a53edf451779af91b035a248bcd34311eec16bf6a2f34d973b22877bd8c Sep 11 15:09:58.631865 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.631840 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-28bqs" Sep 11 15:09:58.646312 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.646259 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-28bqs" podStartSLOduration=1.29110892 podStartE2EDuration="2.646238239s" totalImagesPullingTime="1.355129319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:09:58.645354823 +0000 UTC m=+92.877279254" watchObservedRunningTime="2026-09-11 15:09:58.646238239 +0000 UTC m=+92.878162670" Sep 11 15:09:58.744940 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:58.744829 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865d6dbf9f-6nhcq"] Sep 11 15:09:58.751857 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:09:58.750296 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a7d6d6_a976_4f52_b05f_5cc7d28ce379.slice/crio-38dc9e2255a020516bb21b8781477667efe07eb2c133d68c74ca7f43c680fb8f WatchSource:0}: Error finding container 38dc9e2255a020516bb21b8781477667efe07eb2c133d68c74ca7f43c680fb8f: Status 404 returned error can't find the container with id 38dc9e2255a020516bb21b8781477667efe07eb2c133d68c74ca7f43c680fb8f Sep 11 15:09:59.401921 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.401893 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:59.459525 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.459492 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7"] Sep 11 15:09:59.462949 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.462918 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:09:59.468223 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.467723 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lbckc\"" Sep 11 15:09:59.468223 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.468033 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 15:09:59.468223 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.468078 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 11 15:09:59.469409 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.468708 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 15:09:59.469409 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.468982 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 11 15:09:59.469409 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.469220 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 15:09:59.479388 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.479283 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7"] Sep 11 15:09:59.547771 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.547703 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:09:59.632824 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.632606 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8xm5j" event={"ID":"69574f6d-1b8f-4d63-b24d-119756a0267d","Type":"ContainerStarted","Data":"b8fea6b7bc0c2b02d0700fac712baf0fec3febb9b2537bbabd383147d5fe38bc"} Sep 11 15:09:59.632824 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.632689 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8xm5j" event={"ID":"69574f6d-1b8f-4d63-b24d-119756a0267d","Type":"ContainerStarted","Data":"ed7e61298e4025c1a986d3c8c238af56c98148943a067d355e958701b7db7612"} Sep 11 15:09:59.632824 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.632721 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8xm5j" event={"ID":"69574f6d-1b8f-4d63-b24d-119756a0267d","Type":"ContainerStarted","Data":"935d7a53edf451779af91b035a248bcd34311eec16bf6a2f34d973b22877bd8c"} Sep 11 15:09:59.634508 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.634480 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865d6dbf9f-6nhcq" event={"ID":"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379","Type":"ContainerStarted","Data":"38dc9e2255a020516bb21b8781477667efe07eb2c133d68c74ca7f43c680fb8f"} Sep 11 15:09:59.639207 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.639180 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx44m\" (UniqueName: \"kubernetes.io/projected/ea38a4c7-6e87-4e46-ac15-a530c206f519-kube-api-access-xx44m\") pod \"prometheus-operator-68bcd7f987-cpqn7\" (UID: \"ea38a4c7-6e87-4e46-ac15-a530c206f519\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:09:59.639345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.639265 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea38a4c7-6e87-4e46-ac15-a530c206f519-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-cpqn7\" (UID: \"ea38a4c7-6e87-4e46-ac15-a530c206f519\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:09:59.639345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.639317 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea38a4c7-6e87-4e46-ac15-a530c206f519-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cpqn7\" (UID: \"ea38a4c7-6e87-4e46-ac15-a530c206f519\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:09:59.639457 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.639352 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea38a4c7-6e87-4e46-ac15-a530c206f519-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-cpqn7\" (UID: \"ea38a4c7-6e87-4e46-ac15-a530c206f519\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:09:59.740259 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.740228 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea38a4c7-6e87-4e46-ac15-a530c206f519-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-cpqn7\" (UID: \"ea38a4c7-6e87-4e46-ac15-a530c206f519\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:09:59.740448 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.740293 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea38a4c7-6e87-4e46-ac15-a530c206f519-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cpqn7\" (UID: \"ea38a4c7-6e87-4e46-ac15-a530c206f519\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:09:59.740448 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.740345 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea38a4c7-6e87-4e46-ac15-a530c206f519-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-cpqn7\" (UID: \"ea38a4c7-6e87-4e46-ac15-a530c206f519\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:09:59.740448 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.740394 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx44m\" (UniqueName: \"kubernetes.io/projected/ea38a4c7-6e87-4e46-ac15-a530c206f519-kube-api-access-xx44m\") pod \"prometheus-operator-68bcd7f987-cpqn7\" (UID: \"ea38a4c7-6e87-4e46-ac15-a530c206f519\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:09:59.740783 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:59.740763 2558 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 11 15:09:59.740871 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:09:59.740830 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea38a4c7-6e87-4e46-ac15-a530c206f519-prometheus-operator-tls podName:ea38a4c7-6e87-4e46-ac15-a530c206f519 nodeName:}" failed. No retries permitted until 2026-09-11 15:10:00.240811966 +0000 UTC m=+94.472736389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ea38a4c7-6e87-4e46-ac15-a530c206f519-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-cpqn7" (UID: "ea38a4c7-6e87-4e46-ac15-a530c206f519") : secret "prometheus-operator-tls" not found Sep 11 15:09:59.741493 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.741443 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea38a4c7-6e87-4e46-ac15-a530c206f519-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-cpqn7\" (UID: \"ea38a4c7-6e87-4e46-ac15-a530c206f519\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:09:59.744185 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.744163 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea38a4c7-6e87-4e46-ac15-a530c206f519-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-cpqn7\" (UID: \"ea38a4c7-6e87-4e46-ac15-a530c206f519\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:09:59.752924 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:09:59.752764 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx44m\" (UniqueName: \"kubernetes.io/projected/ea38a4c7-6e87-4e46-ac15-a530c206f519-kube-api-access-xx44m\") pod \"prometheus-operator-68bcd7f987-cpqn7\" (UID: \"ea38a4c7-6e87-4e46-ac15-a530c206f519\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:10:00.092765 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:00.092670 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:00.245647 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:00.245082 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea38a4c7-6e87-4e46-ac15-a530c206f519-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cpqn7\" (UID: \"ea38a4c7-6e87-4e46-ac15-a530c206f519\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:10:00.253314 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:00.253256 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ea38a4c7-6e87-4e46-ac15-a530c206f519-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cpqn7\" (UID: \"ea38a4c7-6e87-4e46-ac15-a530c206f519\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:10:00.375770 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:00.375593 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" Sep 11 15:10:00.679953 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:00.679822 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7"] Sep 11 15:10:01.056894 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:10:01.056816 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea38a4c7_6e87_4e46_ac15_a530c206f519.slice/crio-6c33790298fd87cc6d74054d1e91b0f07fccd4a5013dfc1662c44313aa9c048a WatchSource:0}: Error finding container 6c33790298fd87cc6d74054d1e91b0f07fccd4a5013dfc1662c44313aa9c048a: Status 404 returned error can't find the container with id 6c33790298fd87cc6d74054d1e91b0f07fccd4a5013dfc1662c44313aa9c048a Sep 11 15:10:01.628710 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:01.628672 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:01.651120 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:01.651077 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8xm5j" event={"ID":"69574f6d-1b8f-4d63-b24d-119756a0267d","Type":"ContainerStarted","Data":"2b6ea5aada73673f09ae9f534f1e61c36c4afae32dff553fdb21a324287c569f"} Sep 11 15:10:01.653045 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:01.653005 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" event={"ID":"ea38a4c7-6e87-4e46-ac15-a530c206f519","Type":"ContainerStarted","Data":"6c33790298fd87cc6d74054d1e91b0f07fccd4a5013dfc1662c44313aa9c048a"} Sep 11 15:10:01.675631 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:01.675134 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8xm5j" podStartSLOduration=33.345426722 podStartE2EDuration="35.675113153s" totalImagesPullingTime="2.329686431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:09:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:01.673129755 +0000 UTC m=+95.905054187" watchObservedRunningTime="2026-09-11 15:10:01.675113153 +0000 UTC m=+95.907037586" Sep 11 15:10:01.733818 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:01.733777 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:03.628280 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:03.628238 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:03.724525 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:03.724487 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:03.964992 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:03.964952 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:04.153624 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:04.153580 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:04.408774 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:04.408725 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:04.582110 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:04.582069 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:04.582327 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:04.582299 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:10:04.582446 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:04.582394 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:10:04.584957 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:04.584928 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/584cffdd-40bc-4fd9-bd8f-fb92dd86978e-metrics-tls\") pod \"dns-default-tngb6\" (UID: \"584cffdd-40bc-4fd9-bd8f-fb92dd86978e\") " pod="openshift-dns/dns-default-tngb6" Sep 11 15:10:04.585048 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:04.585031 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls\") pod \"image-registry-7fddbb4949-qnxrp\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:10:04.815681 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:04.815608 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tz2qn\"" Sep 11 15:10:04.823732 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:04.823705 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:10:04.833100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:04.833077 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zldr7\"" Sep 11 15:10:04.840302 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:04.840280 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tngb6" Sep 11 15:10:05.013063 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:05.013033 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:05.119244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:05.119163 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:05.207808 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:05.207775 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:08.043634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:08.043594 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:08.158444 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:08.158400 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:09.547314 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:09.547285 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qcg9p" Sep 11 15:10:10.300192 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.300106 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tngb6"] Sep 11 15:10:10.309193 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.309163 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fddbb4949-qnxrp"] Sep 11 15:10:10.370614 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:10:10.370583 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584cffdd_40bc_4fd9_bd8f_fb92dd86978e.slice/crio-7707e521ed950ff90ee11d11289408297fb40e4d2a4fa2438f0ce17e7a6168fe WatchSource:0}: Error finding container 7707e521ed950ff90ee11d11289408297fb40e4d2a4fa2438f0ce17e7a6168fe: Status 404 returned error can't find the container with id 7707e521ed950ff90ee11d11289408297fb40e4d2a4fa2438f0ce17e7a6168fe Sep 11 15:10:10.371228 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:10:10.371180 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d09616_cdc7_41d0_9a02_f888fb741327.slice/crio-3070cdb74064ee110cd4212a34b866e22b4ba4742491f017dfbd01a71d15f97c WatchSource:0}: Error finding container 3070cdb74064ee110cd4212a34b866e22b4ba4742491f017dfbd01a71d15f97c: Status 404 returned error can't find the container with id 3070cdb74064ee110cd4212a34b866e22b4ba4742491f017dfbd01a71d15f97c Sep 11 15:10:10.689960 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.689307 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-52rkt" event={"ID":"9fa10168-da0c-4e3f-aaf6-73db67dd6227","Type":"ContainerStarted","Data":"b9a3f15e067e1cf674b9cb085e306d9c20e196db6de17f705e42b76078848278"} Sep 11 15:10:10.690428 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.690203 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-52rkt" Sep 11 15:10:10.691510 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.691454 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865d6dbf9f-6nhcq" event={"ID":"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379","Type":"ContainerStarted","Data":"296647bf3d77fa174522506f38ff9f5fa3ef6eff4eceabcd26d5f6543cd247d5"} Sep 11 15:10:10.692794 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.692766 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tngb6" event={"ID":"584cffdd-40bc-4fd9-bd8f-fb92dd86978e","Type":"ContainerStarted","Data":"7707e521ed950ff90ee11d11289408297fb40e4d2a4fa2438f0ce17e7a6168fe"} Sep 11 15:10:10.694588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.694518 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" event={"ID":"20d09616-cdc7-41d0-9a02-f888fb741327","Type":"ContainerStarted","Data":"9357f3e86812aba4bf652e49522d5813a49b5fa77514b5920f49ceb690e5718c"} Sep 11 15:10:10.694588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.694575 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" event={"ID":"20d09616-cdc7-41d0-9a02-f888fb741327","Type":"ContainerStarted","Data":"3070cdb74064ee110cd4212a34b866e22b4ba4742491f017dfbd01a71d15f97c"} Sep 11 15:10:10.694768 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.694684 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:10:10.696857 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.696774 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" event={"ID":"ea38a4c7-6e87-4e46-ac15-a530c206f519","Type":"ContainerStarted","Data":"1be5df70cbf66f65bee9d0da22e7285e4b8fbc084fd13e485285c29d8e9d7de0"} Sep 11 15:10:10.696857 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.696825 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" event={"ID":"ea38a4c7-6e87-4e46-ac15-a530c206f519","Type":"ContainerStarted","Data":"fcd7ff1ee70d1cdea53e9e45fc77cc4e532fd0cb796971f0bc308e9c8f0061e5"} Sep 11 15:10:10.708200 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.708176 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-52rkt" Sep 11 15:10:10.713332 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.713286 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-52rkt" podStartSLOduration=2.694318198 podStartE2EDuration="20.713271801s" totalImagesPullingTime="18.018953603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:10.712323346 +0000 UTC m=+104.944247777" watchObservedRunningTime="2026-09-11 15:10:10.713271801 +0000 UTC m=+104.945196229" Sep 11 15:10:10.736711 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.736612 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-865d6dbf9f-6nhcq" podStartSLOduration=1.464774995 podStartE2EDuration="12.736592239s" totalImagesPullingTime="11.271817244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:10.735094705 +0000 UTC m=+104.967019168" watchObservedRunningTime="2026-09-11 15:10:10.736592239 +0000 UTC m=+104.968516673" Sep 11 15:10:10.744235 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.744115 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:10.756103 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.756043 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cpqn7" podStartSLOduration=2.432069416 podStartE2EDuration="11.756021506s" totalImagesPullingTime="9.32395209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:10.755232249 +0000 UTC m=+104.987156677" watchObservedRunningTime="2026-09-11 15:10:10.756021506 +0000 UTC m=+104.987945940" Sep 11 15:10:10.780860 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.780802 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" podStartSLOduration=103.780783644 podStartE2EDuration="1m43.780783644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:10.780454481 +0000 UTC m=+105.012378935" watchObservedRunningTime="2026-09-11 15:10:10.780783644 +0000 UTC m=+105.012708075" Sep 11 15:10:10.971684 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:10.971594 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fddbb4949-qnxrp"] Sep 11 15:10:11.066548 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:11.066516 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:11.198882 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:11.198847 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:11.300396 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:11.300300 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:12.595655 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:12.595613 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:12.712583 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:12.712471 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:12.817174 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:12.817101 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:12.924190 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:12.924141 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-48qcs"] Sep 11 15:10:12.933057 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:12.932982 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:12.949019 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:12.949000 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:12.951748 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:12.951709 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 15:10:12.952155 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:12.952007 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 15:10:12.952155 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:12.952048 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 15:10:12.952919 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:12.952901 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zcpm2\"" Sep 11 15:10:13.055857 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.055821 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdlzc\" (UniqueName: \"kubernetes.io/projected/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-kube-api-access-wdlzc\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.056007 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.055907 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-metrics-client-ca\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.056007 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.055940 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.056007 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.055967 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-wtmp\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.056170 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.056024 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-root\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.056170 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.056052 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-accelerators-collector-config\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.056170 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.056086 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-sys\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.056170 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.056113 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-tls\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.056170 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.056153 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-textfile\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.062515 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.062415 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:13.152428 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.152335 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.158448 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-metrics-client-ca\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.158497 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.158547 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-wtmp\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.158637 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-root\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.158669 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-accelerators-collector-config\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.158712 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-sys\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.158769 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-tls\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.158806 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-textfile\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.158844 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdlzc\" (UniqueName: \"kubernetes.io/projected/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-kube-api-access-wdlzc\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:10:13.159718 2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:10:13.159872 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-tls podName:ac7a7cd7-2c65-443a-85a2-b7da044b1a53 nodeName:}" failed. No retries permitted until 2026-09-11 15:10:13.65983736 +0000 UTC m=+107.891761796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-tls") pod "node-exporter-48qcs" (UID: "ac7a7cd7-2c65-443a-85a2-b7da044b1a53") : secret "node-exporter-tls" not found Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.159906 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-metrics-client-ca\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.160229 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-sys\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.160246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.160242 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-textfile\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.160934 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.160381 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-wtmp\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.160934 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.160442 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-root\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.172895 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.170213 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-accelerators-collector-config\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.178322 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.177185 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdlzc\" (UniqueName: \"kubernetes.io/projected/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-kube-api-access-wdlzc\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.178322 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.177494 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.261031 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.258916 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:13.354775 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.354723 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:13.455867 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.455779 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:13.554382 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.554347 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:13.655442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.655399 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:13.663800 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.663771 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-tls\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:13.663979 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:10:13.663955 2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 15:10:13.664066 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:10:13.664030 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-tls podName:ac7a7cd7-2c65-443a-85a2-b7da044b1a53 nodeName:}" failed. No retries permitted until 2026-09-11 15:10:14.664010322 +0000 UTC m=+108.895934743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-tls") pod "node-exporter-48qcs" (UID: "ac7a7cd7-2c65-443a-85a2-b7da044b1a53") : secret "node-exporter-tls" not found Sep 11 15:10:13.712154 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.712078 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tngb6" event={"ID":"584cffdd-40bc-4fd9-bd8f-fb92dd86978e","Type":"ContainerStarted","Data":"688e310e7e7ff8eb0894c59db66a593a2f1d8842b6f5a07bd76bfb71c16118e0"} Sep 11 15:10:13.712290 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.712170 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tngb6" event={"ID":"584cffdd-40bc-4fd9-bd8f-fb92dd86978e","Type":"ContainerStarted","Data":"597cd1327dbdc4383e9b6ec8e46e0be8d08c119e790abe20ed262a251865abae"} Sep 11 15:10:13.712290 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.712223 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tngb6" Sep 11 15:10:13.735897 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.735840 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tngb6" podStartSLOduration=71.182643306 podStartE2EDuration="1m13.73581861s" totalImagesPullingTime="2.553175304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:13.733952273 +0000 UTC m=+107.965876707" watchObservedRunningTime="2026-09-11 15:10:13.73581861 +0000 UTC m=+107.967743040" Sep 11 15:10:13.850942 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.850908 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:13.933373 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:13.933329 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:14.033180 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:14.033101 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-865d6dbf9f-6nhcq"] Sep 11 15:10:14.035983 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:14.035939 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:14.672424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:14.672387 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-tls\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:14.675054 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:14.675030 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac7a7cd7-2c65-443a-85a2-b7da044b1a53-node-exporter-tls\") pod \"node-exporter-48qcs\" (UID: \"ac7a7cd7-2c65-443a-85a2-b7da044b1a53\") " pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:14.761715 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:14.761680 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-48qcs" Sep 11 15:10:14.789124 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:10:14.789091 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7a7cd7_2c65_443a_85a2_b7da044b1a53.slice/crio-d7526d02d1d4892dd557c923c8762810bcc4870b6c1869c9223c6b5fe6bbb190 WatchSource:0}: Error finding container d7526d02d1d4892dd557c923c8762810bcc4870b6c1869c9223c6b5fe6bbb190: Status 404 returned error can't find the container with id d7526d02d1d4892dd557c923c8762810bcc4870b6c1869c9223c6b5fe6bbb190 Sep 11 15:10:15.723472 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:15.723389 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48qcs" event={"ID":"ac7a7cd7-2c65-443a-85a2-b7da044b1a53","Type":"ContainerStarted","Data":"d7526d02d1d4892dd557c923c8762810bcc4870b6c1869c9223c6b5fe6bbb190"} Sep 11 15:10:15.819582 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:15.819521 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:15.922157 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:15.922119 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:16.266559 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:16.266515 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:16.731255 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:16.731220 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48qcs" event={"ID":"ac7a7cd7-2c65-443a-85a2-b7da044b1a53","Type":"ContainerStarted","Data":"79385c666b9d4c91db3a88691a7e0d2faea70a2865683ff83588c57ed1655c86"} Sep 11 15:10:16.840766 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:16.840630 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:17.391044 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.391006 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-755d4b68f4-4vhnv"] Sep 11 15:10:17.415993 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.415958 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-755d4b68f4-4vhnv"] Sep 11 15:10:17.416206 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.416075 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.418838 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.418811 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 15:10:17.418974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.418904 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 11 15:10:17.419966 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.419942 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-et5njbrjiscl6\"" Sep 11 15:10:17.420089 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.419988 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 11 15:10:17.420290 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.420270 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 11 15:10:17.420876 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.420482 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-m7ql8\"" Sep 11 15:10:17.422402 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.422374 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:17.498712 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.498662 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6grq5\" (UniqueName: \"kubernetes.io/projected/f23ecfa6-77ec-4278-b534-84916a0d33e1-kube-api-access-6grq5\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.498900 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.498800 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f23ecfa6-77ec-4278-b534-84916a0d33e1-secret-metrics-server-tls\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.498900 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.498835 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f23ecfa6-77ec-4278-b534-84916a0d33e1-secret-metrics-server-client-certs\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.499021 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.498908 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f23ecfa6-77ec-4278-b534-84916a0d33e1-audit-log\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.499021 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.498940 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f23ecfa6-77ec-4278-b534-84916a0d33e1-metrics-server-audit-profiles\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.499021 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.498964 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f23ecfa6-77ec-4278-b534-84916a0d33e1-client-ca-bundle\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.499021 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.499010 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f23ecfa6-77ec-4278-b534-84916a0d33e1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.524524 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.524489 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:17.599829 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.599795 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f23ecfa6-77ec-4278-b534-84916a0d33e1-secret-metrics-server-tls\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.600002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.599840 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f23ecfa6-77ec-4278-b534-84916a0d33e1-secret-metrics-server-client-certs\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.600002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.599903 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f23ecfa6-77ec-4278-b534-84916a0d33e1-audit-log\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.600002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.599937 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f23ecfa6-77ec-4278-b534-84916a0d33e1-metrics-server-audit-profiles\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.600002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.599959 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f23ecfa6-77ec-4278-b534-84916a0d33e1-client-ca-bundle\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.600208 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.600004 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f23ecfa6-77ec-4278-b534-84916a0d33e1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.600208 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.600054 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grq5\" (UniqueName: \"kubernetes.io/projected/f23ecfa6-77ec-4278-b534-84916a0d33e1-kube-api-access-6grq5\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.600377 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.600348 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f23ecfa6-77ec-4278-b534-84916a0d33e1-audit-log\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.601303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.601005 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f23ecfa6-77ec-4278-b534-84916a0d33e1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.601303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.601200 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f23ecfa6-77ec-4278-b534-84916a0d33e1-metrics-server-audit-profiles\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.603591 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.603404 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f23ecfa6-77ec-4278-b534-84916a0d33e1-client-ca-bundle\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.603591 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.603558 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f23ecfa6-77ec-4278-b534-84916a0d33e1-secret-metrics-server-tls\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.604299 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.604272 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f23ecfa6-77ec-4278-b534-84916a0d33e1-secret-metrics-server-client-certs\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.615543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.615519 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6grq5\" (UniqueName: \"kubernetes.io/projected/f23ecfa6-77ec-4278-b534-84916a0d33e1-kube-api-access-6grq5\") pod \"metrics-server-755d4b68f4-4vhnv\" (UID: \"f23ecfa6-77ec-4278-b534-84916a0d33e1\") " pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.631699 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.631670 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:17.721757 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.721705 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:17.727714 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.727686 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:17.737578 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.737552 2558 generic.go:356] "Generic (PLEG): container finished" podID="ac7a7cd7-2c65-443a-85a2-b7da044b1a53" containerID="79385c666b9d4c91db3a88691a7e0d2faea70a2865683ff83588c57ed1655c86" exitCode=0 Sep 11 15:10:17.738010 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:17.737640 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48qcs" event={"ID":"ac7a7cd7-2c65-443a-85a2-b7da044b1a53","Type":"ContainerDied","Data":"79385c666b9d4c91db3a88691a7e0d2faea70a2865683ff83588c57ed1655c86"} Sep 11 15:10:18.041544 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.041481 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-755d4b68f4-4vhnv"] Sep 11 15:10:18.044398 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:10:18.044371 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23ecfa6_77ec_4278_b534_84916a0d33e1.slice/crio-7d587623c1e3ce474284e26d5c5d5bd30b5587578fd5dafb9b418b1ad60b6aa3 WatchSource:0}: Error finding container 7d587623c1e3ce474284e26d5c5d5bd30b5587578fd5dafb9b418b1ad60b6aa3: Status 404 returned error can't find the container with id 7d587623c1e3ce474284e26d5c5d5bd30b5587578fd5dafb9b418b1ad60b6aa3 Sep 11 15:10:18.187275 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.187240 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b5d5cd569-9v555"] Sep 11 15:10:18.218855 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.218825 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b5d5cd569-9v555"] Sep 11 15:10:18.219012 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.218963 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.221613 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.221587 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 11 15:10:18.221725 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.221623 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 11 15:10:18.221941 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.221911 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rcmfs\"" Sep 11 15:10:18.221941 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.221924 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 11 15:10:18.222095 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.221944 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 11 15:10:18.222532 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.222366 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 11 15:10:18.228075 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.228050 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 11 15:10:18.248326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.248269 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:18.307954 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.307927 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/661a06de-e120-4fcb-816d-90b4941120e8-secret-telemeter-client\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.308079 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.307957 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/661a06de-e120-4fcb-816d-90b4941120e8-telemeter-client-tls\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.308079 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.308053 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/661a06de-e120-4fcb-816d-90b4941120e8-metrics-client-ca\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.308171 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.308109 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwbcv\" (UniqueName: \"kubernetes.io/projected/661a06de-e120-4fcb-816d-90b4941120e8-kube-api-access-pwbcv\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.308226 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.308173 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/661a06de-e120-4fcb-816d-90b4941120e8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.308226 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.308192 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661a06de-e120-4fcb-816d-90b4941120e8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.308300 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.308262 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/661a06de-e120-4fcb-816d-90b4941120e8-federate-client-tls\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.308300 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.308294 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661a06de-e120-4fcb-816d-90b4941120e8-serving-certs-ca-bundle\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.324892 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.324864 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:18.408799 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.408767 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwbcv\" (UniqueName: \"kubernetes.io/projected/661a06de-e120-4fcb-816d-90b4941120e8-kube-api-access-pwbcv\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.408999 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.408825 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/661a06de-e120-4fcb-816d-90b4941120e8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.408999 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.408855 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661a06de-e120-4fcb-816d-90b4941120e8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.408999 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.408885 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/661a06de-e120-4fcb-816d-90b4941120e8-federate-client-tls\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.408999 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.408914 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661a06de-e120-4fcb-816d-90b4941120e8-serving-certs-ca-bundle\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.408999 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.408963 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/661a06de-e120-4fcb-816d-90b4941120e8-secret-telemeter-client\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.408999 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.408988 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/661a06de-e120-4fcb-816d-90b4941120e8-telemeter-client-tls\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.409299 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.409030 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/661a06de-e120-4fcb-816d-90b4941120e8-metrics-client-ca\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.409807 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.409776 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661a06de-e120-4fcb-816d-90b4941120e8-serving-certs-ca-bundle\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.409807 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.409794 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/661a06de-e120-4fcb-816d-90b4941120e8-metrics-client-ca\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.410347 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.410324 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661a06de-e120-4fcb-816d-90b4941120e8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.412362 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.412343 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/661a06de-e120-4fcb-816d-90b4941120e8-secret-telemeter-client\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.412526 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.412491 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/661a06de-e120-4fcb-816d-90b4941120e8-federate-client-tls\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.412628 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.412507 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/661a06de-e120-4fcb-816d-90b4941120e8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.413903 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.413879 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/661a06de-e120-4fcb-816d-90b4941120e8-telemeter-client-tls\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.417116 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.417097 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwbcv\" (UniqueName: \"kubernetes.io/projected/661a06de-e120-4fcb-816d-90b4941120e8-kube-api-access-pwbcv\") pod \"telemeter-client-5b5d5cd569-9v555\" (UID: \"661a06de-e120-4fcb-816d-90b4941120e8\") " pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.463575 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.463544 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:10:18.530423 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.530355 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" Sep 11 15:10:18.744929 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.744889 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48qcs" event={"ID":"ac7a7cd7-2c65-443a-85a2-b7da044b1a53","Type":"ContainerStarted","Data":"114d40d7f205995fd09c035478804f26ace69508df7019f6368eb777e98d6f93"} Sep 11 15:10:18.745435 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.744997 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48qcs" event={"ID":"ac7a7cd7-2c65-443a-85a2-b7da044b1a53","Type":"ContainerStarted","Data":"f305ca9f1cf196f41495b77a1416f8f7757d44c3e9cec6f5caf5cb50e6885456"} Sep 11 15:10:18.746393 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.746369 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" event={"ID":"f23ecfa6-77ec-4278-b534-84916a0d33e1","Type":"ContainerStarted","Data":"7d587623c1e3ce474284e26d5c5d5bd30b5587578fd5dafb9b418b1ad60b6aa3"} Sep 11 15:10:18.767076 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.767035 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-48qcs" podStartSLOduration=4.939314018 podStartE2EDuration="6.767021153s" totalImagesPullingTime="1.738769914s" totalInitContainerRuntime="88.937221ms" isStatefulPod=true podCreationTimestamp="2026-09-11 15:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:18.765919848 +0000 UTC m=+112.997844291" watchObservedRunningTime="2026-09-11 15:10:18.767021153 +0000 UTC m=+112.998945584" Sep 11 15:10:18.810612 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.810543 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:18.829971 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.829909 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b5d5cd569-9v555"] Sep 11 15:10:18.835392 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:10:18.835357 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661a06de_e120_4fcb_816d_90b4941120e8.slice/crio-ee32f0b1b73045e39a88d6a48f7ecc29ca2dbe430e0f896eb1a468a9d5988d1c WatchSource:0}: Error finding container ee32f0b1b73045e39a88d6a48f7ecc29ca2dbe430e0f896eb1a468a9d5988d1c: Status 404 returned error can't find the container with id ee32f0b1b73045e39a88d6a48f7ecc29ca2dbe430e0f896eb1a468a9d5988d1c Sep 11 15:10:18.909713 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:18.909645 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:19.122666 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:19.122587 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:19.752729 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:19.752689 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" event={"ID":"661a06de-e120-4fcb-816d-90b4941120e8","Type":"ContainerStarted","Data":"ee32f0b1b73045e39a88d6a48f7ecc29ca2dbe430e0f896eb1a468a9d5988d1c"} Sep 11 15:10:21.762153 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:21.762120 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" event={"ID":"661a06de-e120-4fcb-816d-90b4941120e8","Type":"ContainerStarted","Data":"9b3fe205db14d9199b3c0f76417ef97484d5391017c25a4e01ac60dd0ad57a28"} Sep 11 15:10:21.763692 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:21.763660 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" event={"ID":"f23ecfa6-77ec-4278-b534-84916a0d33e1","Type":"ContainerStarted","Data":"81e275a7ecd222b0f1d09f6aa1eb47f7d94a026b30e7a49df8925a0f24b23842"} Sep 11 15:10:21.785791 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:21.785699 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" podStartSLOduration=1.297082199 podStartE2EDuration="4.78568561s" totalImagesPullingTime="3.488603411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:21.78323668 +0000 UTC m=+116.015161120" watchObservedRunningTime="2026-09-11 15:10:21.78568561 +0000 UTC m=+116.017610040" Sep 11 15:10:23.718935 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:23.718904 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tngb6" Sep 11 15:10:23.773102 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:23.773069 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" event={"ID":"661a06de-e120-4fcb-816d-90b4941120e8","Type":"ContainerStarted","Data":"bb135d2d3d6f0b62073252ebf08924fe7c0aabd56520a0ee9bd235e3695478bd"} Sep 11 15:10:23.773261 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:23.773130 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" event={"ID":"661a06de-e120-4fcb-816d-90b4941120e8","Type":"ContainerStarted","Data":"23bfed173357ef4900cc69f593fbaf4567e4eb0631a41429f00e23a5660aa2c2"} Sep 11 15:10:23.797011 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:23.796961 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b5d5cd569-9v555" podStartSLOduration=1.656376877 podStartE2EDuration="5.796944932s" totalImagesPullingTime="4.140568055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:10:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:23.794289223 +0000 UTC m=+118.026213651" watchObservedRunningTime="2026-09-11 15:10:23.796944932 +0000 UTC m=+118.028869362" Sep 11 15:10:25.140561 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:25.140526 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:25.241886 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:25.241857 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:27.672583 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:27.672546 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:27.737176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:27.737154 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:32.711444 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:32.711414 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:10:32.856436 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:32.856405 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:35.277819 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:35.277788 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:35.379345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:35.379315 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:35.442462 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:35.442435 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:36.158839 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:36.158802 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:10:36.161062 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:36.161044 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaf8fe6d-2de1-4a11-9798-5d3e2e83164e-metrics-certs\") pod \"network-metrics-daemon-lkx5k\" (UID: \"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e\") " pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:10:36.384088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:36.384059 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4h8rg\"" Sep 11 15:10:36.392398 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:36.392372 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkx5k" Sep 11 15:10:36.691800 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:36.691772 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lkx5k"] Sep 11 15:10:36.694230 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:10:36.694203 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf8fe6d_2de1_4a11_9798_5d3e2e83164e.slice/crio-fbc3f94a68999cd78bb9f187677866d65798bc77b4bdfb5643313b3f58e73630 WatchSource:0}: Error finding container fbc3f94a68999cd78bb9f187677866d65798bc77b4bdfb5643313b3f58e73630: Status 404 returned error can't find the container with id fbc3f94a68999cd78bb9f187677866d65798bc77b4bdfb5643313b3f58e73630 Sep 11 15:10:36.821186 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:36.821153 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lkx5k" event={"ID":"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e","Type":"ContainerStarted","Data":"fbc3f94a68999cd78bb9f187677866d65798bc77b4bdfb5643313b3f58e73630"} Sep 11 15:10:37.727767 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:37.727704 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" podUID="20d09616-cdc7-41d0-9a02-f888fb741327" containerName="registry" containerID="cri-o://9357f3e86812aba4bf652e49522d5813a49b5fa77514b5920f49ceb690e5718c" gracePeriod=30 Sep 11 15:10:37.728194 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:37.727788 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:37.728194 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:37.727923 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:38.010469 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.010446 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:10:38.176439 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.176407 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwn98\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-kube-api-access-pwn98\") pod \"20d09616-cdc7-41d0-9a02-f888fb741327\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " Sep 11 15:10:38.176608 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.176454 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20d09616-cdc7-41d0-9a02-f888fb741327-image-registry-private-configuration\") pod \"20d09616-cdc7-41d0-9a02-f888fb741327\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " Sep 11 15:10:38.176608 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.176474 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-bound-sa-token\") pod \"20d09616-cdc7-41d0-9a02-f888fb741327\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " Sep 11 15:10:38.176608 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.176498 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20d09616-cdc7-41d0-9a02-f888fb741327-registry-certificates\") pod \"20d09616-cdc7-41d0-9a02-f888fb741327\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " Sep 11 15:10:38.176608 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.176518 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20d09616-cdc7-41d0-9a02-f888fb741327-trusted-ca\") pod \"20d09616-cdc7-41d0-9a02-f888fb741327\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " Sep 11 15:10:38.176608 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.176549 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20d09616-cdc7-41d0-9a02-f888fb741327-installation-pull-secrets\") pod \"20d09616-cdc7-41d0-9a02-f888fb741327\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " Sep 11 15:10:38.176608 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.176568 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls\") pod \"20d09616-cdc7-41d0-9a02-f888fb741327\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " Sep 11 15:10:38.176608 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.176595 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20d09616-cdc7-41d0-9a02-f888fb741327-ca-trust-extracted\") pod \"20d09616-cdc7-41d0-9a02-f888fb741327\" (UID: \"20d09616-cdc7-41d0-9a02-f888fb741327\") " Sep 11 15:10:38.177011 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.176938 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d09616-cdc7-41d0-9a02-f888fb741327-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "20d09616-cdc7-41d0-9a02-f888fb741327" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:10:38.177011 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.176962 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d09616-cdc7-41d0-9a02-f888fb741327-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "20d09616-cdc7-41d0-9a02-f888fb741327" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:10:38.179052 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.179005 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-kube-api-access-pwn98" (OuterVolumeSpecName: "kube-api-access-pwn98") pod "20d09616-cdc7-41d0-9a02-f888fb741327" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327"). InnerVolumeSpecName "kube-api-access-pwn98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:10:38.179191 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.179063 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d09616-cdc7-41d0-9a02-f888fb741327-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "20d09616-cdc7-41d0-9a02-f888fb741327" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:10:38.179264 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.179208 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "20d09616-cdc7-41d0-9a02-f888fb741327" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:10:38.179264 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.179225 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d09616-cdc7-41d0-9a02-f888fb741327-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "20d09616-cdc7-41d0-9a02-f888fb741327" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:10:38.179446 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.179422 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "20d09616-cdc7-41d0-9a02-f888fb741327" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:10:38.185453 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.185424 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d09616-cdc7-41d0-9a02-f888fb741327-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "20d09616-cdc7-41d0-9a02-f888fb741327" (UID: "20d09616-cdc7-41d0-9a02-f888fb741327"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:10:38.277239 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.277217 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwn98\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-kube-api-access-pwn98\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:38.277239 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.277241 2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/20d09616-cdc7-41d0-9a02-f888fb741327-image-registry-private-configuration\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:38.277359 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.277250 2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-bound-sa-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:38.277359 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.277259 2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/20d09616-cdc7-41d0-9a02-f888fb741327-registry-certificates\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:38.277359 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.277267 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20d09616-cdc7-41d0-9a02-f888fb741327-trusted-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:38.277359 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.277277 2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/20d09616-cdc7-41d0-9a02-f888fb741327-installation-pull-secrets\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:38.277359 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.277286 2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/20d09616-cdc7-41d0-9a02-f888fb741327-registry-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:38.277359 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.277293 2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/20d09616-cdc7-41d0-9a02-f888fb741327-ca-trust-extracted\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:38.830481 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.830441 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lkx5k" event={"ID":"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e","Type":"ContainerStarted","Data":"7321e4d9b1da7144b34a82855ecbce2b37031f7977e2c57717b08580cb273887"} Sep 11 15:10:38.830912 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.830513 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lkx5k" event={"ID":"aaf8fe6d-2de1-4a11-9798-5d3e2e83164e","Type":"ContainerStarted","Data":"48e3f5e664bf3aa0ec699e0904353419148262a45644c7286f9d342dca61cbee"} Sep 11 15:10:38.831843 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.831817 2558 generic.go:356] "Generic (PLEG): container finished" podID="20d09616-cdc7-41d0-9a02-f888fb741327" containerID="9357f3e86812aba4bf652e49522d5813a49b5fa77514b5920f49ceb690e5718c" exitCode=0 Sep 11 15:10:38.831937 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.831855 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" Sep 11 15:10:38.831974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.831858 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" event={"ID":"20d09616-cdc7-41d0-9a02-f888fb741327","Type":"ContainerDied","Data":"9357f3e86812aba4bf652e49522d5813a49b5fa77514b5920f49ceb690e5718c"} Sep 11 15:10:38.832008 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.831979 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fddbb4949-qnxrp" event={"ID":"20d09616-cdc7-41d0-9a02-f888fb741327","Type":"ContainerDied","Data":"3070cdb74064ee110cd4212a34b866e22b4ba4742491f017dfbd01a71d15f97c"} Sep 11 15:10:38.832008 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.832002 2558 scope.go:117] "RemoveContainer" containerID="9357f3e86812aba4bf652e49522d5813a49b5fa77514b5920f49ceb690e5718c" Sep 11 15:10:38.843065 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.843045 2558 scope.go:117] "RemoveContainer" containerID="9357f3e86812aba4bf652e49522d5813a49b5fa77514b5920f49ceb690e5718c" Sep 11 15:10:38.843399 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:10:38.843378 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9357f3e86812aba4bf652e49522d5813a49b5fa77514b5920f49ceb690e5718c\": container with ID starting with 9357f3e86812aba4bf652e49522d5813a49b5fa77514b5920f49ceb690e5718c not found: ID does not exist" containerID="9357f3e86812aba4bf652e49522d5813a49b5fa77514b5920f49ceb690e5718c" Sep 11 15:10:38.843464 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.843406 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9357f3e86812aba4bf652e49522d5813a49b5fa77514b5920f49ceb690e5718c"} err="failed to get container status \"9357f3e86812aba4bf652e49522d5813a49b5fa77514b5920f49ceb690e5718c\": rpc error: code = NotFound desc = could not find container \"9357f3e86812aba4bf652e49522d5813a49b5fa77514b5920f49ceb690e5718c\": container with ID starting with 9357f3e86812aba4bf652e49522d5813a49b5fa77514b5920f49ceb690e5718c not found: ID does not exist" Sep 11 15:10:38.850979 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.850911 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lkx5k" podStartSLOduration=130.364177688 podStartE2EDuration="2m11.850879637s" totalImagesPullingTime="1.486701949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:10:38.849140748 +0000 UTC m=+133.081065177" watchObservedRunningTime="2026-09-11 15:10:38.850879637 +0000 UTC m=+133.082804058" Sep 11 15:10:38.866425 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.866404 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fddbb4949-qnxrp"] Sep 11 15:10:38.871763 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.871725 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fddbb4949-qnxrp"] Sep 11 15:10:38.961729 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:38.961707 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:39.032109 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.032077 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:39.057435 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.057404 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-865d6dbf9f-6nhcq" podUID="d3a7d6d6-a976-4f52-b05f-5cc7d28ce379" containerName="console" containerID="cri-o://296647bf3d77fa174522506f38ff9f5fa3ef6eff4eceabcd26d5f6543cd247d5" gracePeriod=15 Sep 11 15:10:39.338639 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.338583 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:10:39.487820 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.487791 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-service-ca\") pod \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " Sep 11 15:10:39.488019 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.487862 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b94mt\" (UniqueName: \"kubernetes.io/projected/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-kube-api-access-b94mt\") pod \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " Sep 11 15:10:39.488019 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.487891 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-oauth-config\") pod \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " Sep 11 15:10:39.488019 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.487938 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-config\") pod \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " Sep 11 15:10:39.488019 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.487966 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-oauth-serving-cert\") pod \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " Sep 11 15:10:39.488019 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.488010 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-serving-cert\") pod \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\" (UID: \"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379\") " Sep 11 15:10:39.488337 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.488218 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-service-ca" (OuterVolumeSpecName: "service-ca") pod "d3a7d6d6-a976-4f52-b05f-5cc7d28ce379" (UID: "d3a7d6d6-a976-4f52-b05f-5cc7d28ce379"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:10:39.488426 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.488401 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-config" (OuterVolumeSpecName: "console-config") pod "d3a7d6d6-a976-4f52-b05f-5cc7d28ce379" (UID: "d3a7d6d6-a976-4f52-b05f-5cc7d28ce379"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:10:39.488478 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.488405 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d3a7d6d6-a976-4f52-b05f-5cc7d28ce379" (UID: "d3a7d6d6-a976-4f52-b05f-5cc7d28ce379"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:10:39.490178 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.490146 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d3a7d6d6-a976-4f52-b05f-5cc7d28ce379" (UID: "d3a7d6d6-a976-4f52-b05f-5cc7d28ce379"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:10:39.490286 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.490210 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d3a7d6d6-a976-4f52-b05f-5cc7d28ce379" (UID: "d3a7d6d6-a976-4f52-b05f-5cc7d28ce379"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:10:39.490286 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.490268 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-kube-api-access-b94mt" (OuterVolumeSpecName: "kube-api-access-b94mt") pod "d3a7d6d6-a976-4f52-b05f-5cc7d28ce379" (UID: "d3a7d6d6-a976-4f52-b05f-5cc7d28ce379"). InnerVolumeSpecName "kube-api-access-b94mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:10:39.589069 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.589001 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-serving-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:39.589069 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.589028 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:39.589069 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.589041 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b94mt\" (UniqueName: \"kubernetes.io/projected/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-kube-api-access-b94mt\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:39.589069 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.589054 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-oauth-config\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:39.589069 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.589067 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-console-config\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:39.589292 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.589080 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379-oauth-serving-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:10:39.836679 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.836644 2558 generic.go:356] "Generic (PLEG): container finished" podID="d3a7d6d6-a976-4f52-b05f-5cc7d28ce379" containerID="296647bf3d77fa174522506f38ff9f5fa3ef6eff4eceabcd26d5f6543cd247d5" exitCode=0 Sep 11 15:10:39.837107 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.836687 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865d6dbf9f-6nhcq" Sep 11 15:10:39.837107 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.836732 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865d6dbf9f-6nhcq" event={"ID":"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379","Type":"ContainerDied","Data":"296647bf3d77fa174522506f38ff9f5fa3ef6eff4eceabcd26d5f6543cd247d5"} Sep 11 15:10:39.837107 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.836797 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865d6dbf9f-6nhcq" event={"ID":"d3a7d6d6-a976-4f52-b05f-5cc7d28ce379","Type":"ContainerDied","Data":"38dc9e2255a020516bb21b8781477667efe07eb2c133d68c74ca7f43c680fb8f"} Sep 11 15:10:39.837107 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.836827 2558 scope.go:117] "RemoveContainer" containerID="296647bf3d77fa174522506f38ff9f5fa3ef6eff4eceabcd26d5f6543cd247d5" Sep 11 15:10:39.844944 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.844799 2558 scope.go:117] "RemoveContainer" containerID="296647bf3d77fa174522506f38ff9f5fa3ef6eff4eceabcd26d5f6543cd247d5" Sep 11 15:10:39.845120 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:10:39.845100 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296647bf3d77fa174522506f38ff9f5fa3ef6eff4eceabcd26d5f6543cd247d5\": container with ID starting with 296647bf3d77fa174522506f38ff9f5fa3ef6eff4eceabcd26d5f6543cd247d5 not found: ID does not exist" containerID="296647bf3d77fa174522506f38ff9f5fa3ef6eff4eceabcd26d5f6543cd247d5" Sep 11 15:10:39.845194 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.845128 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296647bf3d77fa174522506f38ff9f5fa3ef6eff4eceabcd26d5f6543cd247d5"} err="failed to get container status \"296647bf3d77fa174522506f38ff9f5fa3ef6eff4eceabcd26d5f6543cd247d5\": rpc error: code = NotFound desc = could not find container \"296647bf3d77fa174522506f38ff9f5fa3ef6eff4eceabcd26d5f6543cd247d5\": container with ID starting with 296647bf3d77fa174522506f38ff9f5fa3ef6eff4eceabcd26d5f6543cd247d5 not found: ID does not exist" Sep 11 15:10:39.857027 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.857003 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-865d6dbf9f-6nhcq"] Sep 11 15:10:39.860849 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:39.860829 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-865d6dbf9f-6nhcq"] Sep 11 15:10:40.263502 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:40.263467 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20d09616-cdc7-41d0-9a02-f888fb741327" path="/var/lib/kubelet/pods/20d09616-cdc7-41d0-9a02-f888fb741327/volumes" Sep 11 15:10:40.263870 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:40.263855 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3a7d6d6-a976-4f52-b05f-5cc7d28ce379" path="/var/lib/kubelet/pods/d3a7d6d6-a976-4f52-b05f-5cc7d28ce379/volumes" Sep 11 15:10:55.902437 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:55.902401 2558 generic.go:356] "Generic (PLEG): container finished" podID="598ebdf6-3cba-4a77-b9ab-b8edaee3db5f" containerID="807049c4b8a1efda11898607d30d4dcc0304973596e23e57c1e07490977c4cdd" exitCode=0 Sep 11 15:10:55.902866 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:55.902461 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" event={"ID":"598ebdf6-3cba-4a77-b9ab-b8edaee3db5f","Type":"ContainerDied","Data":"807049c4b8a1efda11898607d30d4dcc0304973596e23e57c1e07490977c4cdd"} Sep 11 15:10:55.902866 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:55.902731 2558 scope.go:117] "RemoveContainer" containerID="807049c4b8a1efda11898607d30d4dcc0304973596e23e57c1e07490977c4cdd" Sep 11 15:10:56.908475 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:56.908439 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xdjlq" event={"ID":"598ebdf6-3cba-4a77-b9ab-b8edaee3db5f","Type":"ContainerStarted","Data":"4f6d0718d131c3b1074ab566a211deb777a10a8eafa20005bf5ccb70ef2895fe"} Sep 11 15:10:57.683177 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:57.683136 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:10:57.733330 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:57.733308 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:57.737028 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:57.737006 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-755d4b68f4-4vhnv" Sep 11 15:10:57.747005 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:10:57.746959 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:00.624898 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:00.624851 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:29.023501 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:29.023462 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:29.100869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:29.100839 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:35.618750 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:35.618717 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:40.522057 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.522026 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dc79d45f7-lng6r"] Sep 11 15:11:40.522418 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.522284 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20d09616-cdc7-41d0-9a02-f888fb741327" containerName="registry" Sep 11 15:11:40.522418 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.522295 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="20d09616-cdc7-41d0-9a02-f888fb741327" containerName="registry" Sep 11 15:11:40.522418 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.522309 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a7d6d6-a976-4f52-b05f-5cc7d28ce379" containerName="console" Sep 11 15:11:40.522418 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.522315 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a7d6d6-a976-4f52-b05f-5cc7d28ce379" containerName="console" Sep 11 15:11:40.522418 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.522362 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a7d6d6-a976-4f52-b05f-5cc7d28ce379" containerName="console" Sep 11 15:11:40.522418 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.522371 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="20d09616-cdc7-41d0-9a02-f888fb741327" containerName="registry" Sep 11 15:11:40.526240 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.526219 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.529124 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.529103 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 15:11:40.530104 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.530078 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 15:11:40.530104 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.530098 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 15:11:40.530278 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.530171 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 15:11:40.530278 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.530193 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 15:11:40.530378 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.530366 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wnkdc\"" Sep 11 15:11:40.535226 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.534316 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 15:11:40.541496 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.541477 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dc79d45f7-lng6r"] Sep 11 15:11:40.584489 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.584461 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:40.661692 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.661661 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:40.671397 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.671374 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-trusted-ca-bundle\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.671475 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.671403 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-config\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.671475 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.671422 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-oauth-serving-cert\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.671475 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.671449 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z8j6\" (UniqueName: \"kubernetes.io/projected/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-kube-api-access-9z8j6\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.671634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.671539 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-oauth-config\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.671634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.671579 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-serving-cert\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.671732 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.671675 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-service-ca\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.772507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.772432 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-oauth-config\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.772507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.772467 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-serving-cert\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.772764 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.772511 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-service-ca\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.772764 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.772536 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-trusted-ca-bundle\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.772764 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.772553 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-config\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.772764 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.772568 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-oauth-serving-cert\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.772764 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.772584 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z8j6\" (UniqueName: \"kubernetes.io/projected/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-kube-api-access-9z8j6\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.773366 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.773341 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-config\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.773451 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.773431 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-trusted-ca-bundle\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.773492 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.773440 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-oauth-serving-cert\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.773550 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.773527 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-service-ca\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.774953 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.774934 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-oauth-config\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.775047 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.775030 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-serving-cert\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.790306 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.790289 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z8j6\" (UniqueName: \"kubernetes.io/projected/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-kube-api-access-9z8j6\") pod \"console-6dc79d45f7-lng6r\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.839886 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.839859 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:40.947016 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:40.946988 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:41.028007 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:41.027943 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:41.113047 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:41.113022 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dc79d45f7-lng6r"] Sep 11 15:11:41.114874 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:11:41.114849 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0f4757_4dc5_4d5a_a938_ed26766e8e0e.slice/crio-f2c1fc30e25c6a6c53800f9280ab87db0d4d997b01bfdb64676d83daf600a788 WatchSource:0}: Error finding container f2c1fc30e25c6a6c53800f9280ab87db0d4d997b01bfdb64676d83daf600a788: Status 404 returned error can't find the container with id f2c1fc30e25c6a6c53800f9280ab87db0d4d997b01bfdb64676d83daf600a788 Sep 11 15:11:42.077290 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:42.077254 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc79d45f7-lng6r" event={"ID":"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e","Type":"ContainerStarted","Data":"8bee888840dc80d68d1d874171eaba9c1e82bd76113c8f32d34c1fcb3b8df5ae"} Sep 11 15:11:42.077640 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:42.077300 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc79d45f7-lng6r" event={"ID":"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e","Type":"ContainerStarted","Data":"f2c1fc30e25c6a6c53800f9280ab87db0d4d997b01bfdb64676d83daf600a788"} Sep 11 15:11:42.099163 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:42.099121 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dc79d45f7-lng6r" podStartSLOduration=2.099105626 podStartE2EDuration="2.099105626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:11:42.09864408 +0000 UTC m=+196.330568535" watchObservedRunningTime="2026-09-11 15:11:42.099105626 +0000 UTC m=+196.331030058" Sep 11 15:11:50.840767 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:50.840704 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:50.841215 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:50.840796 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:50.845444 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:50.845425 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:51.112425 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:51.112351 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:11:51.290367 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:51.290332 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:51.386128 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:51.386041 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:51.486921 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:51.486889 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:57.675934 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:57.675899 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:57.776955 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:57.776920 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:11:57.836196 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:11:57.836167 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:12:22.636117 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:22.636080 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:12:22.692038 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:22.692005 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:12:38.772713 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:38.772683 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:12:50.137720 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:50.137647 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:12:50.236278 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:50.236244 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:12:51.132517 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.132484 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr"] Sep 11 15:12:51.135704 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.135683 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" Sep 11 15:12:51.143034 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.138658 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 15:12:51.143034 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.139336 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 15:12:51.143034 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.139560 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sztdq\"" Sep 11 15:12:51.147134 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.147110 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr"] Sep 11 15:12:51.307697 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.307669 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee185aa5-9c72-4a4a-8f29-460c3df43675-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr\" (UID: \"ee185aa5-9c72-4a4a-8f29-460c3df43675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" Sep 11 15:12:51.307914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.307718 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee185aa5-9c72-4a4a-8f29-460c3df43675-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr\" (UID: \"ee185aa5-9c72-4a4a-8f29-460c3df43675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" Sep 11 15:12:51.307914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.307774 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z586h\" (UniqueName: \"kubernetes.io/projected/ee185aa5-9c72-4a4a-8f29-460c3df43675-kube-api-access-z586h\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr\" (UID: \"ee185aa5-9c72-4a4a-8f29-460c3df43675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" Sep 11 15:12:51.408405 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.408374 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee185aa5-9c72-4a4a-8f29-460c3df43675-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr\" (UID: \"ee185aa5-9c72-4a4a-8f29-460c3df43675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" Sep 11 15:12:51.408573 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.408421 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee185aa5-9c72-4a4a-8f29-460c3df43675-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr\" (UID: \"ee185aa5-9c72-4a4a-8f29-460c3df43675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" Sep 11 15:12:51.408573 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.408542 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z586h\" (UniqueName: \"kubernetes.io/projected/ee185aa5-9c72-4a4a-8f29-460c3df43675-kube-api-access-z586h\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr\" (UID: \"ee185aa5-9c72-4a4a-8f29-460c3df43675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" Sep 11 15:12:51.408772 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.408729 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee185aa5-9c72-4a4a-8f29-460c3df43675-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr\" (UID: \"ee185aa5-9c72-4a4a-8f29-460c3df43675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" Sep 11 15:12:51.408848 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.408774 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee185aa5-9c72-4a4a-8f29-460c3df43675-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr\" (UID: \"ee185aa5-9c72-4a4a-8f29-460c3df43675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" Sep 11 15:12:51.417668 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.417645 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z586h\" (UniqueName: \"kubernetes.io/projected/ee185aa5-9c72-4a4a-8f29-460c3df43675-kube-api-access-z586h\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr\" (UID: \"ee185aa5-9c72-4a4a-8f29-460c3df43675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" Sep 11 15:12:51.450486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.450465 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" Sep 11 15:12:51.719956 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:51.719926 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr"] Sep 11 15:12:51.722684 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:12:51.722657 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee185aa5_9c72_4a4a_8f29_460c3df43675.slice/crio-666bdef8f07380f1e65889b42efa933b37eb640ccfb9228e4613e8f5ad9b4f07 WatchSource:0}: Error finding container 666bdef8f07380f1e65889b42efa933b37eb640ccfb9228e4613e8f5ad9b4f07: Status 404 returned error can't find the container with id 666bdef8f07380f1e65889b42efa933b37eb640ccfb9228e4613e8f5ad9b4f07 Sep 11 15:12:52.324057 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:52.324019 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" event={"ID":"ee185aa5-9c72-4a4a-8f29-460c3df43675","Type":"ContainerStarted","Data":"666bdef8f07380f1e65889b42efa933b37eb640ccfb9228e4613e8f5ad9b4f07"} Sep 11 15:12:58.351287 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:58.351249 2558 generic.go:356] "Generic (PLEG): container finished" podID="ee185aa5-9c72-4a4a-8f29-460c3df43675" containerID="ebded1c4f67e903f1fbe8311e3f29d32d9bd73b014b040fc34704b9d83c64270" exitCode=0 Sep 11 15:12:58.351644 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:12:58.351325 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" event={"ID":"ee185aa5-9c72-4a4a-8f29-460c3df43675","Type":"ContainerDied","Data":"ebded1c4f67e903f1fbe8311e3f29d32d9bd73b014b040fc34704b9d83c64270"} Sep 11 15:13:01.364031 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:01.363997 2558 generic.go:356] "Generic (PLEG): container finished" podID="ee185aa5-9c72-4a4a-8f29-460c3df43675" containerID="8c9c1559046df5af12dcb760dcccda0959fc8493b625269d8c4dcec3ee25ef4f" exitCode=0 Sep 11 15:13:01.364411 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:01.364087 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" event={"ID":"ee185aa5-9c72-4a4a-8f29-460c3df43675","Type":"ContainerDied","Data":"8c9c1559046df5af12dcb760dcccda0959fc8493b625269d8c4dcec3ee25ef4f"} Sep 11 15:13:04.158777 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:04.158716 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:10.400041 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:10.400008 2558 generic.go:356] "Generic (PLEG): container finished" podID="ee185aa5-9c72-4a4a-8f29-460c3df43675" containerID="4a057419dce1f5172969f32ec070e308932dd58fa36fc52d3f7df5752ec82eb8" exitCode=0 Sep 11 15:13:10.400413 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:10.400049 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" event={"ID":"ee185aa5-9c72-4a4a-8f29-460c3df43675","Type":"ContainerDied","Data":"4a057419dce1f5172969f32ec070e308932dd58fa36fc52d3f7df5752ec82eb8"} Sep 11 15:13:11.533640 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:11.533615 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" Sep 11 15:13:11.574478 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:11.574443 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee185aa5-9c72-4a4a-8f29-460c3df43675-bundle\") pod \"ee185aa5-9c72-4a4a-8f29-460c3df43675\" (UID: \"ee185aa5-9c72-4a4a-8f29-460c3df43675\") " Sep 11 15:13:11.574660 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:11.574497 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee185aa5-9c72-4a4a-8f29-460c3df43675-util\") pod \"ee185aa5-9c72-4a4a-8f29-460c3df43675\" (UID: \"ee185aa5-9c72-4a4a-8f29-460c3df43675\") " Sep 11 15:13:11.574660 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:11.574560 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z586h\" (UniqueName: \"kubernetes.io/projected/ee185aa5-9c72-4a4a-8f29-460c3df43675-kube-api-access-z586h\") pod \"ee185aa5-9c72-4a4a-8f29-460c3df43675\" (UID: \"ee185aa5-9c72-4a4a-8f29-460c3df43675\") " Sep 11 15:13:11.575242 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:11.575220 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee185aa5-9c72-4a4a-8f29-460c3df43675-bundle" (OuterVolumeSpecName: "bundle") pod "ee185aa5-9c72-4a4a-8f29-460c3df43675" (UID: "ee185aa5-9c72-4a4a-8f29-460c3df43675"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:13:11.576811 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:11.576787 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee185aa5-9c72-4a4a-8f29-460c3df43675-kube-api-access-z586h" (OuterVolumeSpecName: "kube-api-access-z586h") pod "ee185aa5-9c72-4a4a-8f29-460c3df43675" (UID: "ee185aa5-9c72-4a4a-8f29-460c3df43675"). InnerVolumeSpecName "kube-api-access-z586h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:13:11.578651 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:11.578628 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee185aa5-9c72-4a4a-8f29-460c3df43675-util" (OuterVolumeSpecName: "util") pod "ee185aa5-9c72-4a4a-8f29-460c3df43675" (UID: "ee185aa5-9c72-4a4a-8f29-460c3df43675"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:13:11.675658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:11.675558 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee185aa5-9c72-4a4a-8f29-460c3df43675-util\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:13:11.675658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:11.675602 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z586h\" (UniqueName: \"kubernetes.io/projected/ee185aa5-9c72-4a4a-8f29-460c3df43675-kube-api-access-z586h\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:13:11.675658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:11.675614 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee185aa5-9c72-4a4a-8f29-460c3df43675-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:13:12.410095 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:12.410064 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" Sep 11 15:13:12.410259 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:12.410071 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dnmhr" event={"ID":"ee185aa5-9c72-4a4a-8f29-460c3df43675","Type":"ContainerDied","Data":"666bdef8f07380f1e65889b42efa933b37eb640ccfb9228e4613e8f5ad9b4f07"} Sep 11 15:13:12.410259 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:12.410183 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="666bdef8f07380f1e65889b42efa933b37eb640ccfb9228e4613e8f5ad9b4f07" Sep 11 15:13:17.186089 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:17.186054 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:18.006733 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:18.006697 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:19.160868 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.160836 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9"] Sep 11 15:13:19.161241 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.161095 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee185aa5-9c72-4a4a-8f29-460c3df43675" containerName="util" Sep 11 15:13:19.161241 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.161105 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee185aa5-9c72-4a4a-8f29-460c3df43675" containerName="util" Sep 11 15:13:19.161241 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.161118 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee185aa5-9c72-4a4a-8f29-460c3df43675" containerName="extract" Sep 11 15:13:19.161241 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.161124 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee185aa5-9c72-4a4a-8f29-460c3df43675" containerName="extract" Sep 11 15:13:19.161241 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.161133 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee185aa5-9c72-4a4a-8f29-460c3df43675" containerName="pull" Sep 11 15:13:19.161241 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.161139 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee185aa5-9c72-4a4a-8f29-460c3df43675" containerName="pull" Sep 11 15:13:19.161241 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.161193 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee185aa5-9c72-4a4a-8f29-460c3df43675" containerName="extract" Sep 11 15:13:19.163889 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.163874 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9" Sep 11 15:13:19.166602 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.166580 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 11 15:13:19.166723 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.166664 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:13:19.166808 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.166724 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rpcc2\"" Sep 11 15:13:19.177449 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.177429 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9"] Sep 11 15:13:19.230340 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.230314 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60694eeb-f33c-4659-a95c-d7de7262ce7d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9n2c9\" (UID: \"60694eeb-f33c-4659-a95c-d7de7262ce7d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9" Sep 11 15:13:19.230475 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.230345 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bwrn\" (UniqueName: \"kubernetes.io/projected/60694eeb-f33c-4659-a95c-d7de7262ce7d-kube-api-access-4bwrn\") pod \"cert-manager-operator-controller-manager-55c444bf86-9n2c9\" (UID: \"60694eeb-f33c-4659-a95c-d7de7262ce7d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9" Sep 11 15:13:19.331057 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.331011 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bwrn\" (UniqueName: \"kubernetes.io/projected/60694eeb-f33c-4659-a95c-d7de7262ce7d-kube-api-access-4bwrn\") pod \"cert-manager-operator-controller-manager-55c444bf86-9n2c9\" (UID: \"60694eeb-f33c-4659-a95c-d7de7262ce7d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9" Sep 11 15:13:19.331220 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.331111 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60694eeb-f33c-4659-a95c-d7de7262ce7d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9n2c9\" (UID: \"60694eeb-f33c-4659-a95c-d7de7262ce7d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9" Sep 11 15:13:19.331437 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.331420 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60694eeb-f33c-4659-a95c-d7de7262ce7d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9n2c9\" (UID: \"60694eeb-f33c-4659-a95c-d7de7262ce7d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9" Sep 11 15:13:19.346545 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.346503 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bwrn\" (UniqueName: \"kubernetes.io/projected/60694eeb-f33c-4659-a95c-d7de7262ce7d-kube-api-access-4bwrn\") pod \"cert-manager-operator-controller-manager-55c444bf86-9n2c9\" (UID: \"60694eeb-f33c-4659-a95c-d7de7262ce7d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9" Sep 11 15:13:19.473374 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.473285 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9" Sep 11 15:13:19.752646 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:19.752620 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9"] Sep 11 15:13:19.755139 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:13:19.755104 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60694eeb_f33c_4659_a95c_d7de7262ce7d.slice/crio-480669efd8823d18a3d0d5c99dd54a47b639c8084dbfe1da4b679c16cd20b630 WatchSource:0}: Error finding container 480669efd8823d18a3d0d5c99dd54a47b639c8084dbfe1da4b679c16cd20b630: Status 404 returned error can't find the container with id 480669efd8823d18a3d0d5c99dd54a47b639c8084dbfe1da4b679c16cd20b630 Sep 11 15:13:20.443301 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:20.443266 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9" event={"ID":"60694eeb-f33c-4659-a95c-d7de7262ce7d","Type":"ContainerStarted","Data":"480669efd8823d18a3d0d5c99dd54a47b639c8084dbfe1da4b679c16cd20b630"} Sep 11 15:13:20.807685 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:20.807592 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:20.909833 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:20.909792 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:22.151050 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:22.151013 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:22.246922 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:22.246890 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:22.454895 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:22.454808 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9" event={"ID":"60694eeb-f33c-4659-a95c-d7de7262ce7d","Type":"ContainerStarted","Data":"c16bc6c9802d68c6e57223dae5b8895a1c0abc8cfbfee11ef01191f338b1aae5"} Sep 11 15:13:22.479662 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:22.479602 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9n2c9" podStartSLOduration=1.704017791 podStartE2EDuration="3.479585852s" totalImagesPullingTime="1.775568061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:13:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:13:22.479135165 +0000 UTC m=+296.711059597" watchObservedRunningTime="2026-09-11 15:13:22.479585852 +0000 UTC m=+296.711510282" Sep 11 15:13:22.989212 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:22.989176 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:23.186518 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.186476 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:23.500148 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.500116 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8"] Sep 11 15:13:23.503553 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.503534 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" Sep 11 15:13:23.506121 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.506098 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 15:13:23.506231 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.506212 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 15:13:23.507206 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.507187 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sztdq\"" Sep 11 15:13:23.513775 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.513754 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8"] Sep 11 15:13:23.567376 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.567335 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bda78af2-f25d-4a92-ba8e-55c27812f888-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8\" (UID: \"bda78af2-f25d-4a92-ba8e-55c27812f888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" Sep 11 15:13:23.567546 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.567396 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djfkr\" (UniqueName: \"kubernetes.io/projected/bda78af2-f25d-4a92-ba8e-55c27812f888-kube-api-access-djfkr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8\" (UID: \"bda78af2-f25d-4a92-ba8e-55c27812f888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" Sep 11 15:13:23.567546 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.567422 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bda78af2-f25d-4a92-ba8e-55c27812f888-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8\" (UID: \"bda78af2-f25d-4a92-ba8e-55c27812f888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" Sep 11 15:13:23.668502 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.668469 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bda78af2-f25d-4a92-ba8e-55c27812f888-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8\" (UID: \"bda78af2-f25d-4a92-ba8e-55c27812f888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" Sep 11 15:13:23.668671 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.668511 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djfkr\" (UniqueName: \"kubernetes.io/projected/bda78af2-f25d-4a92-ba8e-55c27812f888-kube-api-access-djfkr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8\" (UID: \"bda78af2-f25d-4a92-ba8e-55c27812f888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" Sep 11 15:13:23.668671 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.668534 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bda78af2-f25d-4a92-ba8e-55c27812f888-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8\" (UID: \"bda78af2-f25d-4a92-ba8e-55c27812f888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" Sep 11 15:13:23.668892 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.668870 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bda78af2-f25d-4a92-ba8e-55c27812f888-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8\" (UID: \"bda78af2-f25d-4a92-ba8e-55c27812f888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" Sep 11 15:13:23.668934 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.668902 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bda78af2-f25d-4a92-ba8e-55c27812f888-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8\" (UID: \"bda78af2-f25d-4a92-ba8e-55c27812f888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" Sep 11 15:13:23.680269 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.680235 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djfkr\" (UniqueName: \"kubernetes.io/projected/bda78af2-f25d-4a92-ba8e-55c27812f888-kube-api-access-djfkr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8\" (UID: \"bda78af2-f25d-4a92-ba8e-55c27812f888\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" Sep 11 15:13:23.812699 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:23.812607 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" Sep 11 15:13:24.091012 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.090956 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8"] Sep 11 15:13:24.093572 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:13:24.093540 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda78af2_f25d_4a92_ba8e_55c27812f888.slice/crio-81c10d82f7d1ccca652d3bdaf9077b174795ece0132b879adad9c8a3954fd170 WatchSource:0}: Error finding container 81c10d82f7d1ccca652d3bdaf9077b174795ece0132b879adad9c8a3954fd170: Status 404 returned error can't find the container with id 81c10d82f7d1ccca652d3bdaf9077b174795ece0132b879adad9c8a3954fd170 Sep 11 15:13:24.169477 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.169455 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:24.464599 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.464565 2558 generic.go:356] "Generic (PLEG): container finished" podID="bda78af2-f25d-4a92-ba8e-55c27812f888" containerID="1a1cb437f0358df165ab01e88947627f7f97593dae5f584d0584f274fddc3ab4" exitCode=0 Sep 11 15:13:24.464994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.464617 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" event={"ID":"bda78af2-f25d-4a92-ba8e-55c27812f888","Type":"ContainerDied","Data":"1a1cb437f0358df165ab01e88947627f7f97593dae5f584d0584f274fddc3ab4"} Sep 11 15:13:24.464994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.464671 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" event={"ID":"bda78af2-f25d-4a92-ba8e-55c27812f888","Type":"ContainerStarted","Data":"81c10d82f7d1ccca652d3bdaf9077b174795ece0132b879adad9c8a3954fd170"} Sep 11 15:13:24.532096 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.532065 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:24.920055 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.920023 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6shsr"] Sep 11 15:13:24.923171 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.923148 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6shsr" Sep 11 15:13:24.929280 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.925517 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-59dmf\"" Sep 11 15:13:24.929280 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.925843 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 15:13:24.929280 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.926107 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 15:13:24.932336 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.932311 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6shsr"] Sep 11 15:13:24.979643 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.979609 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbgk4\" (UniqueName: \"kubernetes.io/projected/0bb7b28e-2e90-41e2-9103-7c0daf52a2e3-kube-api-access-pbgk4\") pod \"cert-manager-cainjector-559cdd8c87-6shsr\" (UID: \"0bb7b28e-2e90-41e2-9103-7c0daf52a2e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6shsr" Sep 11 15:13:24.979643 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:24.979649 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bb7b28e-2e90-41e2-9103-7c0daf52a2e3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6shsr\" (UID: \"0bb7b28e-2e90-41e2-9103-7c0daf52a2e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6shsr" Sep 11 15:13:25.080588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:25.080556 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbgk4\" (UniqueName: \"kubernetes.io/projected/0bb7b28e-2e90-41e2-9103-7c0daf52a2e3-kube-api-access-pbgk4\") pod \"cert-manager-cainjector-559cdd8c87-6shsr\" (UID: \"0bb7b28e-2e90-41e2-9103-7c0daf52a2e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6shsr" Sep 11 15:13:25.080588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:25.080604 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bb7b28e-2e90-41e2-9103-7c0daf52a2e3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6shsr\" (UID: \"0bb7b28e-2e90-41e2-9103-7c0daf52a2e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6shsr" Sep 11 15:13:25.089074 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:25.089042 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bb7b28e-2e90-41e2-9103-7c0daf52a2e3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6shsr\" (UID: \"0bb7b28e-2e90-41e2-9103-7c0daf52a2e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6shsr" Sep 11 15:13:25.089207 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:25.089170 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbgk4\" (UniqueName: \"kubernetes.io/projected/0bb7b28e-2e90-41e2-9103-7c0daf52a2e3-kube-api-access-pbgk4\") pod \"cert-manager-cainjector-559cdd8c87-6shsr\" (UID: \"0bb7b28e-2e90-41e2-9103-7c0daf52a2e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6shsr" Sep 11 15:13:25.238482 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:25.238397 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6shsr" Sep 11 15:13:25.515256 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:25.515175 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6shsr"] Sep 11 15:13:25.518505 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:13:25.518476 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bb7b28e_2e90_41e2_9103_7c0daf52a2e3.slice/crio-fcadacd16dec36cf8a69db42fae759401d8af604a28315b304dc6f6a96546a0c WatchSource:0}: Error finding container fcadacd16dec36cf8a69db42fae759401d8af604a28315b304dc6f6a96546a0c: Status 404 returned error can't find the container with id fcadacd16dec36cf8a69db42fae759401d8af604a28315b304dc6f6a96546a0c Sep 11 15:13:26.387569 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:26.387430 2558 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 15:13:26.478460 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:26.478436 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6shsr" event={"ID":"0bb7b28e-2e90-41e2-9103-7c0daf52a2e3","Type":"ContainerStarted","Data":"fcadacd16dec36cf8a69db42fae759401d8af604a28315b304dc6f6a96546a0c"} Sep 11 15:13:27.485515 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:27.485477 2558 generic.go:356] "Generic (PLEG): container finished" podID="bda78af2-f25d-4a92-ba8e-55c27812f888" containerID="5374f7c227434d79bf5469fc694503b0bdfddbd31ab59af385afd9079165941c" exitCode=0 Sep 11 15:13:27.485515 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:27.485524 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" event={"ID":"bda78af2-f25d-4a92-ba8e-55c27812f888","Type":"ContainerDied","Data":"5374f7c227434d79bf5469fc694503b0bdfddbd31ab59af385afd9079165941c"} Sep 11 15:13:27.587667 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:27.587629 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:27.687434 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:27.687405 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:28.491914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:28.491826 2558 generic.go:356] "Generic (PLEG): container finished" podID="bda78af2-f25d-4a92-ba8e-55c27812f888" containerID="2878d7112802762d08bdaa91546df9d47f3dff3e6b085e8b27fb4545c3746ecb" exitCode=0 Sep 11 15:13:28.492316 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:28.491919 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" event={"ID":"bda78af2-f25d-4a92-ba8e-55c27812f888","Type":"ContainerDied","Data":"2878d7112802762d08bdaa91546df9d47f3dff3e6b085e8b27fb4545c3746ecb"} Sep 11 15:13:28.493486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:28.493463 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6shsr" event={"ID":"0bb7b28e-2e90-41e2-9103-7c0daf52a2e3","Type":"ContainerStarted","Data":"b395738aa0bae8b4768dc0d9545772f638390f1264d1aadfcab769700636f6fc"} Sep 11 15:13:28.537727 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:28.537681 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6shsr" podStartSLOduration=1.768041355 podStartE2EDuration="4.537666571s" totalImagesPullingTime="2.769625216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:13:28.536453631 +0000 UTC m=+302.768378063" watchObservedRunningTime="2026-09-11 15:13:28.537666571 +0000 UTC m=+302.769591002" Sep 11 15:13:29.503305 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.503274 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zntql"] Sep 11 15:13:29.506674 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.506654 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zntql" Sep 11 15:13:29.509098 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.509081 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5ldvj\"" Sep 11 15:13:29.517646 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.517625 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zntql"] Sep 11 15:13:29.620856 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.620824 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3efa1af-cadc-44b8-9003-47bd69a579e0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zntql\" (UID: \"f3efa1af-cadc-44b8-9003-47bd69a579e0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zntql" Sep 11 15:13:29.620980 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.620963 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfm4q\" (UniqueName: \"kubernetes.io/projected/f3efa1af-cadc-44b8-9003-47bd69a579e0-kube-api-access-vfm4q\") pod \"cert-manager-webhook-6486ffd997-zntql\" (UID: \"f3efa1af-cadc-44b8-9003-47bd69a579e0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zntql" Sep 11 15:13:29.625763 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.625732 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" Sep 11 15:13:29.721886 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.721857 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djfkr\" (UniqueName: \"kubernetes.io/projected/bda78af2-f25d-4a92-ba8e-55c27812f888-kube-api-access-djfkr\") pod \"bda78af2-f25d-4a92-ba8e-55c27812f888\" (UID: \"bda78af2-f25d-4a92-ba8e-55c27812f888\") " Sep 11 15:13:29.722016 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.721904 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bda78af2-f25d-4a92-ba8e-55c27812f888-bundle\") pod \"bda78af2-f25d-4a92-ba8e-55c27812f888\" (UID: \"bda78af2-f25d-4a92-ba8e-55c27812f888\") " Sep 11 15:13:29.722016 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.721932 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bda78af2-f25d-4a92-ba8e-55c27812f888-util\") pod \"bda78af2-f25d-4a92-ba8e-55c27812f888\" (UID: \"bda78af2-f25d-4a92-ba8e-55c27812f888\") " Sep 11 15:13:29.722085 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.722045 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfm4q\" (UniqueName: \"kubernetes.io/projected/f3efa1af-cadc-44b8-9003-47bd69a579e0-kube-api-access-vfm4q\") pod \"cert-manager-webhook-6486ffd997-zntql\" (UID: \"f3efa1af-cadc-44b8-9003-47bd69a579e0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zntql" Sep 11 15:13:29.722137 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.722121 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3efa1af-cadc-44b8-9003-47bd69a579e0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zntql\" (UID: \"f3efa1af-cadc-44b8-9003-47bd69a579e0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zntql" Sep 11 15:13:29.722432 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.722310 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda78af2-f25d-4a92-ba8e-55c27812f888-bundle" (OuterVolumeSpecName: "bundle") pod "bda78af2-f25d-4a92-ba8e-55c27812f888" (UID: "bda78af2-f25d-4a92-ba8e-55c27812f888"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:13:29.724299 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.724278 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda78af2-f25d-4a92-ba8e-55c27812f888-kube-api-access-djfkr" (OuterVolumeSpecName: "kube-api-access-djfkr") pod "bda78af2-f25d-4a92-ba8e-55c27812f888" (UID: "bda78af2-f25d-4a92-ba8e-55c27812f888"). InnerVolumeSpecName "kube-api-access-djfkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:13:29.727665 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.727645 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda78af2-f25d-4a92-ba8e-55c27812f888-util" (OuterVolumeSpecName: "util") pod "bda78af2-f25d-4a92-ba8e-55c27812f888" (UID: "bda78af2-f25d-4a92-ba8e-55c27812f888"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:13:29.730495 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.730472 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3efa1af-cadc-44b8-9003-47bd69a579e0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zntql\" (UID: \"f3efa1af-cadc-44b8-9003-47bd69a579e0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zntql" Sep 11 15:13:29.730684 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.730666 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfm4q\" (UniqueName: \"kubernetes.io/projected/f3efa1af-cadc-44b8-9003-47bd69a579e0-kube-api-access-vfm4q\") pod \"cert-manager-webhook-6486ffd997-zntql\" (UID: \"f3efa1af-cadc-44b8-9003-47bd69a579e0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zntql" Sep 11 15:13:29.816322 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.816263 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zntql" Sep 11 15:13:29.822529 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.822497 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djfkr\" (UniqueName: \"kubernetes.io/projected/bda78af2-f25d-4a92-ba8e-55c27812f888-kube-api-access-djfkr\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:13:29.822529 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.822528 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bda78af2-f25d-4a92-ba8e-55c27812f888-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:13:29.822646 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:29.822541 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bda78af2-f25d-4a92-ba8e-55c27812f888-util\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:13:30.115772 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:30.115734 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zntql"] Sep 11 15:13:30.117983 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:13:30.117955 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3efa1af_cadc_44b8_9003_47bd69a579e0.slice/crio-8e13ff1922c5708aa2b69c99cc3abc8a482cb0e6b32d8e45060f0315054e09b2 WatchSource:0}: Error finding container 8e13ff1922c5708aa2b69c99cc3abc8a482cb0e6b32d8e45060f0315054e09b2: Status 404 returned error can't find the container with id 8e13ff1922c5708aa2b69c99cc3abc8a482cb0e6b32d8e45060f0315054e09b2 Sep 11 15:13:30.477588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:30.477555 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:30.503106 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:30.503076 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" Sep 11 15:13:30.503106 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:30.503088 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqgqc8" event={"ID":"bda78af2-f25d-4a92-ba8e-55c27812f888","Type":"ContainerDied","Data":"81c10d82f7d1ccca652d3bdaf9077b174795ece0132b879adad9c8a3954fd170"} Sep 11 15:13:30.503308 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:30.503137 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81c10d82f7d1ccca652d3bdaf9077b174795ece0132b879adad9c8a3954fd170" Sep 11 15:13:30.504651 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:30.504631 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zntql" event={"ID":"f3efa1af-cadc-44b8-9003-47bd69a579e0","Type":"ContainerStarted","Data":"1e594aff9f9c1c34515cacafaeac251abde8c53e47eefd678323e9f4d377f94a"} Sep 11 15:13:30.505008 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:30.504660 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zntql" event={"ID":"f3efa1af-cadc-44b8-9003-47bd69a579e0","Type":"ContainerStarted","Data":"8e13ff1922c5708aa2b69c99cc3abc8a482cb0e6b32d8e45060f0315054e09b2"} Sep 11 15:13:30.505008 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:30.504759 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zntql" Sep 11 15:13:30.523745 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:30.523699 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-zntql" podStartSLOduration=1.523685746 podStartE2EDuration="1.523685746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:13:30.521972467 +0000 UTC m=+304.753896898" watchObservedRunningTime="2026-09-11 15:13:30.523685746 +0000 UTC m=+304.755610176" Sep 11 15:13:30.577581 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:30.577558 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:33.359424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:33.359389 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:34.002099 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:34.002060 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:34.102545 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:34.102505 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:36.026214 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.026180 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8"] Sep 11 15:13:36.026603 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.026490 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda78af2-f25d-4a92-ba8e-55c27812f888" containerName="util" Sep 11 15:13:36.026603 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.026501 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda78af2-f25d-4a92-ba8e-55c27812f888" containerName="util" Sep 11 15:13:36.026603 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.026529 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda78af2-f25d-4a92-ba8e-55c27812f888" containerName="extract" Sep 11 15:13:36.026603 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.026535 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda78af2-f25d-4a92-ba8e-55c27812f888" containerName="extract" Sep 11 15:13:36.026603 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.026543 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda78af2-f25d-4a92-ba8e-55c27812f888" containerName="pull" Sep 11 15:13:36.026603 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.026549 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda78af2-f25d-4a92-ba8e-55c27812f888" containerName="pull" Sep 11 15:13:36.026603 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.026597 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda78af2-f25d-4a92-ba8e-55c27812f888" containerName="extract" Sep 11 15:13:36.030355 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.030336 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8" Sep 11 15:13:36.032942 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.032920 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:13:36.033056 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.032993 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 15:13:36.033854 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.033837 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-srnhn\"" Sep 11 15:13:36.040847 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.040822 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8"] Sep 11 15:13:36.170534 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.170488 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af099e5-5c97-4000-bfe4-bff57a6b47a8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kmng8\" (UID: \"9af099e5-5c97-4000-bfe4-bff57a6b47a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8" Sep 11 15:13:36.170706 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.170572 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z7cz\" (UniqueName: \"kubernetes.io/projected/9af099e5-5c97-4000-bfe4-bff57a6b47a8-kube-api-access-4z7cz\") pod \"openshift-lws-operator-fd8ccff4c-kmng8\" (UID: \"9af099e5-5c97-4000-bfe4-bff57a6b47a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8" Sep 11 15:13:36.271020 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.270990 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af099e5-5c97-4000-bfe4-bff57a6b47a8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kmng8\" (UID: \"9af099e5-5c97-4000-bfe4-bff57a6b47a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8" Sep 11 15:13:36.271153 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.271047 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z7cz\" (UniqueName: \"kubernetes.io/projected/9af099e5-5c97-4000-bfe4-bff57a6b47a8-kube-api-access-4z7cz\") pod \"openshift-lws-operator-fd8ccff4c-kmng8\" (UID: \"9af099e5-5c97-4000-bfe4-bff57a6b47a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8" Sep 11 15:13:36.271384 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.271366 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af099e5-5c97-4000-bfe4-bff57a6b47a8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kmng8\" (UID: \"9af099e5-5c97-4000-bfe4-bff57a6b47a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8" Sep 11 15:13:36.284159 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.284115 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z7cz\" (UniqueName: \"kubernetes.io/projected/9af099e5-5c97-4000-bfe4-bff57a6b47a8-kube-api-access-4z7cz\") pod \"openshift-lws-operator-fd8ccff4c-kmng8\" (UID: \"9af099e5-5c97-4000-bfe4-bff57a6b47a8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8" Sep 11 15:13:36.339186 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.339163 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8" Sep 11 15:13:36.511382 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.511351 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zntql" Sep 11 15:13:36.609244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.609218 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8"] Sep 11 15:13:36.611585 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:13:36.611558 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9af099e5_5c97_4000_bfe4_bff57a6b47a8.slice/crio-464c1227a76438b3e5525601ed0d5b13836281174e149232cc153b6585a2687c WatchSource:0}: Error finding container 464c1227a76438b3e5525601ed0d5b13836281174e149232cc153b6585a2687c: Status 404 returned error can't find the container with id 464c1227a76438b3e5525601ed0d5b13836281174e149232cc153b6585a2687c Sep 11 15:13:36.613233 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:36.613218 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 15:13:37.534954 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:37.534922 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8" event={"ID":"9af099e5-5c97-4000-bfe4-bff57a6b47a8","Type":"ContainerStarted","Data":"464c1227a76438b3e5525601ed0d5b13836281174e149232cc153b6585a2687c"} Sep 11 15:13:49.592284 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:49.592251 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8" event={"ID":"9af099e5-5c97-4000-bfe4-bff57a6b47a8","Type":"ContainerStarted","Data":"cac9478070b122b02b604636f98cfa0925b02d7a4826897eb0e986fcfc87a594"} Sep 11 15:13:49.607454 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:49.607392 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kmng8" podStartSLOduration=1.680467146 podStartE2EDuration="13.607375699s" totalImagesPullingTime="11.926908553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:13:49.60715347 +0000 UTC m=+323.839077903" watchObservedRunningTime="2026-09-11 15:13:49.607375699 +0000 UTC m=+323.839300131" Sep 11 15:13:50.373968 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:50.373933 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:51.801184 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:51.801150 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:52.227115 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:52.227073 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:53.604182 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.604144 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr"] Sep 11 15:13:53.607752 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.607718 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" Sep 11 15:13:53.610189 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.610168 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 15:13:53.610284 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.610168 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 15:13:53.610867 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.610850 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sztdq\"" Sep 11 15:13:53.616852 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.616831 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr"] Sep 11 15:13:53.713066 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.713035 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/708abde6-630b-440d-bb90-7b647b717652-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr\" (UID: \"708abde6-630b-440d-bb90-7b647b717652\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" Sep 11 15:13:53.713221 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.713089 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4q4\" (UniqueName: \"kubernetes.io/projected/708abde6-630b-440d-bb90-7b647b717652-kube-api-access-9d4q4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr\" (UID: \"708abde6-630b-440d-bb90-7b647b717652\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" Sep 11 15:13:53.713221 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.713162 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/708abde6-630b-440d-bb90-7b647b717652-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr\" (UID: \"708abde6-630b-440d-bb90-7b647b717652\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" Sep 11 15:13:53.814368 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.814328 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/708abde6-630b-440d-bb90-7b647b717652-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr\" (UID: \"708abde6-630b-440d-bb90-7b647b717652\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" Sep 11 15:13:53.814544 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.814391 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4q4\" (UniqueName: \"kubernetes.io/projected/708abde6-630b-440d-bb90-7b647b717652-kube-api-access-9d4q4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr\" (UID: \"708abde6-630b-440d-bb90-7b647b717652\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" Sep 11 15:13:53.814544 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.814422 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/708abde6-630b-440d-bb90-7b647b717652-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr\" (UID: \"708abde6-630b-440d-bb90-7b647b717652\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" Sep 11 15:13:53.814804 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.814781 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/708abde6-630b-440d-bb90-7b647b717652-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr\" (UID: \"708abde6-630b-440d-bb90-7b647b717652\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" Sep 11 15:13:53.814804 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.814801 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/708abde6-630b-440d-bb90-7b647b717652-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr\" (UID: \"708abde6-630b-440d-bb90-7b647b717652\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" Sep 11 15:13:53.822853 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.822825 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4q4\" (UniqueName: \"kubernetes.io/projected/708abde6-630b-440d-bb90-7b647b717652-kube-api-access-9d4q4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr\" (UID: \"708abde6-630b-440d-bb90-7b647b717652\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" Sep 11 15:13:53.917647 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:53.917624 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" Sep 11 15:13:54.187778 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:54.187679 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr"] Sep 11 15:13:54.191420 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:13:54.191393 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708abde6_630b_440d_bb90_7b647b717652.slice/crio-42e534dc801c4cb98db0dcda6a193d82227aeb4bfd502eaba50a4e8efa2d48b7 WatchSource:0}: Error finding container 42e534dc801c4cb98db0dcda6a193d82227aeb4bfd502eaba50a4e8efa2d48b7: Status 404 returned error can't find the container with id 42e534dc801c4cb98db0dcda6a193d82227aeb4bfd502eaba50a4e8efa2d48b7 Sep 11 15:13:54.614168 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:54.614073 2558 generic.go:356] "Generic (PLEG): container finished" podID="708abde6-630b-440d-bb90-7b647b717652" containerID="f01843c4c3a052380fa382b0b58ae0a345c94857a658c26fcab0cd035940bb14" exitCode=0 Sep 11 15:13:54.614580 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:54.614162 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" event={"ID":"708abde6-630b-440d-bb90-7b647b717652","Type":"ContainerDied","Data":"f01843c4c3a052380fa382b0b58ae0a345c94857a658c26fcab0cd035940bb14"} Sep 11 15:13:54.614580 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:54.614210 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" event={"ID":"708abde6-630b-440d-bb90-7b647b717652","Type":"ContainerStarted","Data":"42e534dc801c4cb98db0dcda6a193d82227aeb4bfd502eaba50a4e8efa2d48b7"} Sep 11 15:13:55.620729 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:55.620649 2558 generic.go:356] "Generic (PLEG): container finished" podID="708abde6-630b-440d-bb90-7b647b717652" containerID="f80324019ac23a4c24bce1542049b8c6b419c4ce554981f9e8edad2eef62acca" exitCode=0 Sep 11 15:13:55.620729 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:55.620713 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" event={"ID":"708abde6-630b-440d-bb90-7b647b717652","Type":"ContainerDied","Data":"f80324019ac23a4c24bce1542049b8c6b419c4ce554981f9e8edad2eef62acca"} Sep 11 15:13:56.456055 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.456021 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:56.541543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.541511 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4"] Sep 11 15:13:56.545158 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.545139 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:56.549595 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.549577 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 11 15:13:56.551447 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.551112 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 11 15:13:56.551848 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.551831 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 11 15:13:56.558833 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.558813 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xqpmn\"" Sep 11 15:13:56.565707 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.565685 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4"] Sep 11 15:13:56.582104 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.582073 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:56.626767 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.626721 2558 generic.go:356] "Generic (PLEG): container finished" podID="708abde6-630b-440d-bb90-7b647b717652" containerID="bb4dce2724c7514b7e1eef6a93acc83c03f4a86a9a1b255b93a2d6db86dd588f" exitCode=0 Sep 11 15:13:56.627064 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.626793 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" event={"ID":"708abde6-630b-440d-bb90-7b647b717652","Type":"ContainerDied","Data":"bb4dce2724c7514b7e1eef6a93acc83c03f4a86a9a1b255b93a2d6db86dd588f"} Sep 11 15:13:56.637820 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.637797 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe914ef9-a2c8-4955-ac2e-0dd01e15ee91-cert\") pod \"lws-controller-manager-68f4698bc6-vs7m4\" (UID: \"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:56.637920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.637884 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe914ef9-a2c8-4955-ac2e-0dd01e15ee91-metrics-cert\") pod \"lws-controller-manager-68f4698bc6-vs7m4\" (UID: \"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:56.637969 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.637920 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fe914ef9-a2c8-4955-ac2e-0dd01e15ee91-manager-config\") pod \"lws-controller-manager-68f4698bc6-vs7m4\" (UID: \"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:56.637969 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.637946 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6sxp\" (UniqueName: \"kubernetes.io/projected/fe914ef9-a2c8-4955-ac2e-0dd01e15ee91-kube-api-access-j6sxp\") pod \"lws-controller-manager-68f4698bc6-vs7m4\" (UID: \"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:56.739133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.739055 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6sxp\" (UniqueName: \"kubernetes.io/projected/fe914ef9-a2c8-4955-ac2e-0dd01e15ee91-kube-api-access-j6sxp\") pod \"lws-controller-manager-68f4698bc6-vs7m4\" (UID: \"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:56.739133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.739104 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe914ef9-a2c8-4955-ac2e-0dd01e15ee91-cert\") pod \"lws-controller-manager-68f4698bc6-vs7m4\" (UID: \"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:56.739328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.739168 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe914ef9-a2c8-4955-ac2e-0dd01e15ee91-metrics-cert\") pod \"lws-controller-manager-68f4698bc6-vs7m4\" (UID: \"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:56.739328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.739209 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fe914ef9-a2c8-4955-ac2e-0dd01e15ee91-manager-config\") pod \"lws-controller-manager-68f4698bc6-vs7m4\" (UID: \"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:56.739875 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.739852 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fe914ef9-a2c8-4955-ac2e-0dd01e15ee91-manager-config\") pod \"lws-controller-manager-68f4698bc6-vs7m4\" (UID: \"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:56.741769 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.741722 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe914ef9-a2c8-4955-ac2e-0dd01e15ee91-metrics-cert\") pod \"lws-controller-manager-68f4698bc6-vs7m4\" (UID: \"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:56.741859 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.741724 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe914ef9-a2c8-4955-ac2e-0dd01e15ee91-cert\") pod \"lws-controller-manager-68f4698bc6-vs7m4\" (UID: \"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:56.747880 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.747859 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6sxp\" (UniqueName: \"kubernetes.io/projected/fe914ef9-a2c8-4955-ac2e-0dd01e15ee91-kube-api-access-j6sxp\") pod \"lws-controller-manager-68f4698bc6-vs7m4\" (UID: \"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91\") " pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:56.853717 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:56.853688 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:57.148679 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:57.148645 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4"] Sep 11 15:13:57.150843 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:13:57.150814 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe914ef9_a2c8_4955_ac2e_0dd01e15ee91.slice/crio-12de2aee00bc952ed194de1f5a31c9ee17c0d7b8a6ab44baa978406a28e6cf0b WatchSource:0}: Error finding container 12de2aee00bc952ed194de1f5a31c9ee17c0d7b8a6ab44baa978406a28e6cf0b: Status 404 returned error can't find the container with id 12de2aee00bc952ed194de1f5a31c9ee17c0d7b8a6ab44baa978406a28e6cf0b Sep 11 15:13:57.632279 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:57.632194 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" event={"ID":"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91","Type":"ContainerStarted","Data":"12de2aee00bc952ed194de1f5a31c9ee17c0d7b8a6ab44baa978406a28e6cf0b"} Sep 11 15:13:57.757537 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:57.757510 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" Sep 11 15:13:57.848520 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:57.848490 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d4q4\" (UniqueName: \"kubernetes.io/projected/708abde6-630b-440d-bb90-7b647b717652-kube-api-access-9d4q4\") pod \"708abde6-630b-440d-bb90-7b647b717652\" (UID: \"708abde6-630b-440d-bb90-7b647b717652\") " Sep 11 15:13:57.848689 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:57.848557 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/708abde6-630b-440d-bb90-7b647b717652-util\") pod \"708abde6-630b-440d-bb90-7b647b717652\" (UID: \"708abde6-630b-440d-bb90-7b647b717652\") " Sep 11 15:13:57.848689 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:57.848598 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/708abde6-630b-440d-bb90-7b647b717652-bundle\") pod \"708abde6-630b-440d-bb90-7b647b717652\" (UID: \"708abde6-630b-440d-bb90-7b647b717652\") " Sep 11 15:13:57.849493 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:57.849463 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708abde6-630b-440d-bb90-7b647b717652-bundle" (OuterVolumeSpecName: "bundle") pod "708abde6-630b-440d-bb90-7b647b717652" (UID: "708abde6-630b-440d-bb90-7b647b717652"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:13:57.850731 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:57.850699 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708abde6-630b-440d-bb90-7b647b717652-kube-api-access-9d4q4" (OuterVolumeSpecName: "kube-api-access-9d4q4") pod "708abde6-630b-440d-bb90-7b647b717652" (UID: "708abde6-630b-440d-bb90-7b647b717652"). InnerVolumeSpecName "kube-api-access-9d4q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:13:57.856186 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:57.856159 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708abde6-630b-440d-bb90-7b647b717652-util" (OuterVolumeSpecName: "util") pod "708abde6-630b-440d-bb90-7b647b717652" (UID: "708abde6-630b-440d-bb90-7b647b717652"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:13:57.949328 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:57.949290 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4q4\" (UniqueName: \"kubernetes.io/projected/708abde6-630b-440d-bb90-7b647b717652-kube-api-access-9d4q4\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:13:57.949466 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:57.949329 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/708abde6-630b-440d-bb90-7b647b717652-util\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:13:57.949466 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:57.949347 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/708abde6-630b-440d-bb90-7b647b717652-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:13:58.638848 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:58.638780 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" Sep 11 15:13:58.639208 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:58.638782 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796affjgbr" event={"ID":"708abde6-630b-440d-bb90-7b647b717652","Type":"ContainerDied","Data":"42e534dc801c4cb98db0dcda6a193d82227aeb4bfd502eaba50a4e8efa2d48b7"} Sep 11 15:13:58.639208 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:58.638905 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42e534dc801c4cb98db0dcda6a193d82227aeb4bfd502eaba50a4e8efa2d48b7" Sep 11 15:13:59.171353 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:59.171324 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:59.230564 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:59.230520 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:13:59.644987 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:59.644954 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" event={"ID":"fe914ef9-a2c8-4955-ac2e-0dd01e15ee91","Type":"ContainerStarted","Data":"aba37c723011cba01e786e66b3cc740574572994a78868c3e7e15272ea731d5f"} Sep 11 15:13:59.645369 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:59.645056 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:13:59.661657 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:13:59.661615 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" podStartSLOduration=2.162025829 podStartE2EDuration="3.661601155s" totalImagesPullingTime="1.499575326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:13:59.659849338 +0000 UTC m=+333.891773782" watchObservedRunningTime="2026-09-11 15:13:59.661601155 +0000 UTC m=+333.893525588" Sep 11 15:14:01.116765 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:01.116717 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:01.178637 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:01.178587 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:03.227632 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.227598 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk"] Sep 11 15:14:03.227997 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.227889 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="708abde6-630b-440d-bb90-7b647b717652" containerName="extract" Sep 11 15:14:03.227997 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.227900 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="708abde6-630b-440d-bb90-7b647b717652" containerName="extract" Sep 11 15:14:03.227997 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.227920 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="708abde6-630b-440d-bb90-7b647b717652" containerName="pull" Sep 11 15:14:03.227997 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.227925 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="708abde6-630b-440d-bb90-7b647b717652" containerName="pull" Sep 11 15:14:03.227997 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.227931 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="708abde6-630b-440d-bb90-7b647b717652" containerName="util" Sep 11 15:14:03.227997 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.227936 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="708abde6-630b-440d-bb90-7b647b717652" containerName="util" Sep 11 15:14:03.227997 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.227989 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="708abde6-630b-440d-bb90-7b647b717652" containerName="extract" Sep 11 15:14:03.233018 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.233001 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" Sep 11 15:14:03.235509 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.235489 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sztdq\"" Sep 11 15:14:03.235686 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.235518 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 15:14:03.236477 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.236463 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 15:14:03.241110 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.241091 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk"] Sep 11 15:14:03.395388 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.395355 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs5s5\" (UniqueName: \"kubernetes.io/projected/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-kube-api-access-hs5s5\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk\" (UID: \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" Sep 11 15:14:03.395556 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.395411 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk\" (UID: \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" Sep 11 15:14:03.395556 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.395469 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk\" (UID: \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" Sep 11 15:14:03.495896 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.495827 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk\" (UID: \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" Sep 11 15:14:03.495896 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.495863 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk\" (UID: \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" Sep 11 15:14:03.496042 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.495909 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5s5\" (UniqueName: \"kubernetes.io/projected/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-kube-api-access-hs5s5\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk\" (UID: \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" Sep 11 15:14:03.496160 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.496144 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk\" (UID: \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" Sep 11 15:14:03.496239 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.496217 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk\" (UID: \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" Sep 11 15:14:03.505442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.505420 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5s5\" (UniqueName: \"kubernetes.io/projected/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-kube-api-access-hs5s5\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk\" (UID: \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" Sep 11 15:14:03.542809 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.542789 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" Sep 11 15:14:03.822292 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:03.822249 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk"] Sep 11 15:14:03.824387 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:14:03.824355 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f004368_fcbe_46ce_a3c9_96dc5ed0f7e6.slice/crio-2d6195a7d0242f95a492c6742299e01628c2a8ed0a32b29edfe1278ca31c72fc WatchSource:0}: Error finding container 2d6195a7d0242f95a492c6742299e01628c2a8ed0a32b29edfe1278ca31c72fc: Status 404 returned error can't find the container with id 2d6195a7d0242f95a492c6742299e01628c2a8ed0a32b29edfe1278ca31c72fc Sep 11 15:14:04.317628 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:04.317593 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:04.413754 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:04.413720 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:04.668944 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:04.668910 2558 generic.go:356] "Generic (PLEG): container finished" podID="5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6" containerID="0f081fe5a91a73720f5d6349de8563792cf8668f76fa3b4d628e211186c7e973" exitCode=0 Sep 11 15:14:04.669105 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:04.668997 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" event={"ID":"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6","Type":"ContainerDied","Data":"0f081fe5a91a73720f5d6349de8563792cf8668f76fa3b4d628e211186c7e973"} Sep 11 15:14:04.669105 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:04.669051 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" event={"ID":"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6","Type":"ContainerStarted","Data":"2d6195a7d0242f95a492c6742299e01628c2a8ed0a32b29edfe1278ca31c72fc"} Sep 11 15:14:04.956192 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:04.956119 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:05.147635 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.147606 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:05.247542 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.247471 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:05.299642 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.299613 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm"] Sep 11 15:14:05.303990 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.303967 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.307031 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.307003 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 15:14:05.307149 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.307131 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 15:14:05.307213 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.307160 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gkvb7\"" Sep 11 15:14:05.307213 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.307182 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 15:14:05.307318 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.307297 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 15:14:05.311576 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.311554 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq468\" (UniqueName: \"kubernetes.io/projected/2852e3fd-6050-427d-a82a-9c573aaa21e3-kube-api-access-qq468\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.311682 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.311598 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2852e3fd-6050-427d-a82a-9c573aaa21e3-manifests\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.311762 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.311728 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2852e3fd-6050-427d-a82a-9c573aaa21e3-tmp\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.311829 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.311801 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2852e3fd-6050-427d-a82a-9c573aaa21e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.311912 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.311846 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2852e3fd-6050-427d-a82a-9c573aaa21e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.321689 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.321669 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm"] Sep 11 15:14:05.412392 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.412362 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq468\" (UniqueName: \"kubernetes.io/projected/2852e3fd-6050-427d-a82a-9c573aaa21e3-kube-api-access-qq468\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.412524 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.412408 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2852e3fd-6050-427d-a82a-9c573aaa21e3-manifests\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.412524 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.412467 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2852e3fd-6050-427d-a82a-9c573aaa21e3-tmp\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.412524 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.412487 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2852e3fd-6050-427d-a82a-9c573aaa21e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.412524 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.412520 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2852e3fd-6050-427d-a82a-9c573aaa21e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.412947 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.412920 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2852e3fd-6050-427d-a82a-9c573aaa21e3-manifests\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.415104 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.415068 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2852e3fd-6050-427d-a82a-9c573aaa21e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.415195 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.415109 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2852e3fd-6050-427d-a82a-9c573aaa21e3-tmp\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.415288 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.415265 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2852e3fd-6050-427d-a82a-9c573aaa21e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.423080 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.423061 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq468\" (UniqueName: \"kubernetes.io/projected/2852e3fd-6050-427d-a82a-9c573aaa21e3-kube-api-access-qq468\") pod \"opendatahub-operator-controller-manager-87954dbdf-6jmfm\" (UID: \"2852e3fd-6050-427d-a82a-9c573aaa21e3\") " pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.487922 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.487890 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:05.588803 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.588697 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:05.616867 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.616839 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:05.677633 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.677585 2558 generic.go:356] "Generic (PLEG): container finished" podID="5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6" containerID="7a70f9942040d548e5a555692f25089e3c20b6dde8c68d15d4855b80d5baecb0" exitCode=0 Sep 11 15:14:05.677633 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.677622 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" event={"ID":"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6","Type":"ContainerDied","Data":"7a70f9942040d548e5a555692f25089e3c20b6dde8c68d15d4855b80d5baecb0"} Sep 11 15:14:05.692492 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.692455 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:05.788243 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.788211 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:05.883889 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.883826 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:05.891993 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.891928 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm"] Sep 11 15:14:05.894255 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:14:05.894232 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2852e3fd_6050_427d_a82a_9c573aaa21e3.slice/crio-08a8916e4bfd4f056385d69282a79134e652330b51894e2eaa4c64a33e191ae5 WatchSource:0}: Error finding container 08a8916e4bfd4f056385d69282a79134e652330b51894e2eaa4c64a33e191ae5: Status 404 returned error can't find the container with id 08a8916e4bfd4f056385d69282a79134e652330b51894e2eaa4c64a33e191ae5 Sep 11 15:14:05.951901 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:05.951866 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:06.683847 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:06.683815 2558 generic.go:356] "Generic (PLEG): container finished" podID="5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6" containerID="274204eb47f2a922f65891e9a3f1b752671190c7f8129a7ff792b5c90d96fe55" exitCode=0 Sep 11 15:14:06.684276 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:06.683904 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" event={"ID":"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6","Type":"ContainerDied","Data":"274204eb47f2a922f65891e9a3f1b752671190c7f8129a7ff792b5c90d96fe55"} Sep 11 15:14:06.685236 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:06.685214 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" event={"ID":"2852e3fd-6050-427d-a82a-9c573aaa21e3","Type":"ContainerStarted","Data":"08a8916e4bfd4f056385d69282a79134e652330b51894e2eaa4c64a33e191ae5"} Sep 11 15:14:07.838729 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:07.838694 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" Sep 11 15:14:07.931634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:07.931601 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-bundle\") pod \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\" (UID: \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\") " Sep 11 15:14:07.931815 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:07.931663 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs5s5\" (UniqueName: \"kubernetes.io/projected/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-kube-api-access-hs5s5\") pod \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\" (UID: \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\") " Sep 11 15:14:07.931815 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:07.931717 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-util\") pod \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\" (UID: \"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6\") " Sep 11 15:14:07.932681 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:07.932647 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-bundle" (OuterVolumeSpecName: "bundle") pod "5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6" (UID: "5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:14:07.935029 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:07.934992 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-kube-api-access-hs5s5" (OuterVolumeSpecName: "kube-api-access-hs5s5") pod "5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6" (UID: "5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6"). InnerVolumeSpecName "kube-api-access-hs5s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:14:07.940481 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:07.940453 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-util" (OuterVolumeSpecName: "util") pod "5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6" (UID: "5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:14:08.032443 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:08.032359 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:14:08.032443 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:08.032397 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs5s5\" (UniqueName: \"kubernetes.io/projected/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-kube-api-access-hs5s5\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:14:08.032443 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:08.032415 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6-util\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:14:08.701397 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:08.701364 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" Sep 11 15:14:08.701397 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:08.701383 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0b8ldk" event={"ID":"5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6","Type":"ContainerDied","Data":"2d6195a7d0242f95a492c6742299e01628c2a8ed0a32b29edfe1278ca31c72fc"} Sep 11 15:14:08.701665 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:08.701437 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d6195a7d0242f95a492c6742299e01628c2a8ed0a32b29edfe1278ca31c72fc" Sep 11 15:14:10.652426 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:10.652390 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68f4698bc6-vs7m4" Sep 11 15:14:10.718499 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:10.718455 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" event={"ID":"2852e3fd-6050-427d-a82a-9c573aaa21e3","Type":"ContainerStarted","Data":"83767d8398d67df71a6464411a3f5454eb4c755afce8ffd787b86892dab80ebe"} Sep 11 15:14:11.725125 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:11.725088 2558 generic.go:356] "Generic (PLEG): container finished" podID="2852e3fd-6050-427d-a82a-9c573aaa21e3" containerID="83767d8398d67df71a6464411a3f5454eb4c755afce8ffd787b86892dab80ebe" exitCode=0 Sep 11 15:14:11.725493 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:11.725139 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" event={"ID":"2852e3fd-6050-427d-a82a-9c573aaa21e3","Type":"ContainerDied","Data":"83767d8398d67df71a6464411a3f5454eb4c755afce8ffd787b86892dab80ebe"} Sep 11 15:14:12.732114 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:12.732075 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" event={"ID":"2852e3fd-6050-427d-a82a-9c573aaa21e3","Type":"ContainerStarted","Data":"262850c45061093f068523be3df2872bd4763dc7ef7b3e94b10ea9d934abf3e7"} Sep 11 15:14:12.732597 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:12.732229 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:12.756338 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:12.756246 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" podStartSLOduration=2.499399164 podStartE2EDuration="7.756231072s" totalImagesPullingTime="4.242708988s" totalInitContainerRuntime="1.01412292s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:14:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:14:12.754910237 +0000 UTC m=+346.986834667" watchObservedRunningTime="2026-09-11 15:14:12.756231072 +0000 UTC m=+346.988155760" Sep 11 15:14:18.448450 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:18.448420 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:18.545009 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:18.544980 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:23.739195 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:23.739159 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-87954dbdf-6jmfm" Sep 11 15:14:25.403164 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:25.403136 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:25.503213 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:25.503183 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:25.720792 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:25.720699 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:26.016647 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:26.016576 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:27.172338 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:27.172309 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:27.267721 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:27.267695 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:28.303147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:28.303113 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:36.452303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.452269 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-62ngs"] Sep 11 15:14:36.452655 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.452571 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6" containerName="pull" Sep 11 15:14:36.452655 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.452584 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6" containerName="pull" Sep 11 15:14:36.452655 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.452595 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6" containerName="util" Sep 11 15:14:36.452655 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.452600 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6" containerName="util" Sep 11 15:14:36.452655 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.452609 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6" containerName="extract" Sep 11 15:14:36.452655 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.452616 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6" containerName="extract" Sep 11 15:14:36.452873 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.452669 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f004368-fcbe-46ce-a3c9-96dc5ed0f7e6" containerName="extract" Sep 11 15:14:36.458067 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.458042 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" Sep 11 15:14:36.461259 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.461231 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 11 15:14:36.461358 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.461328 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-45z4d\"" Sep 11 15:14:36.533890 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.533850 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-62ngs"] Sep 11 15:14:36.534084 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.534045 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:36.551148 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.551119 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p77wn\" (UniqueName: \"kubernetes.io/projected/2ecd26b7-a4e6-4ae1-8485-3a619d964390-kube-api-access-p77wn\") pod \"odh-model-controller-7998c67dc4-62ngs\" (UID: \"2ecd26b7-a4e6-4ae1-8485-3a619d964390\") " pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" Sep 11 15:14:36.551255 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.551167 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ecd26b7-a4e6-4ae1-8485-3a619d964390-cert\") pod \"odh-model-controller-7998c67dc4-62ngs\" (UID: \"2ecd26b7-a4e6-4ae1-8485-3a619d964390\") " pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" Sep 11 15:14:36.625915 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.625883 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:36.652142 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.652115 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p77wn\" (UniqueName: \"kubernetes.io/projected/2ecd26b7-a4e6-4ae1-8485-3a619d964390-kube-api-access-p77wn\") pod \"odh-model-controller-7998c67dc4-62ngs\" (UID: \"2ecd26b7-a4e6-4ae1-8485-3a619d964390\") " pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" Sep 11 15:14:36.652238 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.652163 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ecd26b7-a4e6-4ae1-8485-3a619d964390-cert\") pod \"odh-model-controller-7998c67dc4-62ngs\" (UID: \"2ecd26b7-a4e6-4ae1-8485-3a619d964390\") " pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" Sep 11 15:14:36.652284 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:14:36.652270 2558 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 11 15:14:36.652354 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:14:36.652343 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ecd26b7-a4e6-4ae1-8485-3a619d964390-cert podName:2ecd26b7-a4e6-4ae1-8485-3a619d964390 nodeName:}" failed. No retries permitted until 2026-09-11 15:14:37.152322548 +0000 UTC m=+371.384246959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2ecd26b7-a4e6-4ae1-8485-3a619d964390-cert") pod "odh-model-controller-7998c67dc4-62ngs" (UID: "2ecd26b7-a4e6-4ae1-8485-3a619d964390") : secret "odh-model-controller-webhook-cert" not found Sep 11 15:14:36.660643 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.660611 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p77wn\" (UniqueName: \"kubernetes.io/projected/2ecd26b7-a4e6-4ae1-8485-3a619d964390-kube-api-access-p77wn\") pod \"odh-model-controller-7998c67dc4-62ngs\" (UID: \"2ecd26b7-a4e6-4ae1-8485-3a619d964390\") " pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" Sep 11 15:14:36.772226 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.772149 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-s5g46"] Sep 11 15:14:36.776118 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.776102 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" Sep 11 15:14:36.779138 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.779124 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 11 15:14:36.788187 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.788159 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:36.789300 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.789279 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-s5g46"] Sep 11 15:14:36.853518 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.853489 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjbn2\" (UniqueName: \"kubernetes.io/projected/b44c03e3-cd6a-41ee-9233-777271b72b2c-kube-api-access-jjbn2\") pod \"model-serving-api-54d89bd5fc-s5g46\" (UID: \"b44c03e3-cd6a-41ee-9233-777271b72b2c\") " pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" Sep 11 15:14:36.853659 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.853568 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44c03e3-cd6a-41ee-9233-777271b72b2c-tls-certs\") pod \"model-serving-api-54d89bd5fc-s5g46\" (UID: \"b44c03e3-cd6a-41ee-9233-777271b72b2c\") " pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" Sep 11 15:14:36.882702 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.882672 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:36.954943 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.954918 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44c03e3-cd6a-41ee-9233-777271b72b2c-tls-certs\") pod \"model-serving-api-54d89bd5fc-s5g46\" (UID: \"b44c03e3-cd6a-41ee-9233-777271b72b2c\") " pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" Sep 11 15:14:36.955073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.954997 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjbn2\" (UniqueName: \"kubernetes.io/projected/b44c03e3-cd6a-41ee-9233-777271b72b2c-kube-api-access-jjbn2\") pod \"model-serving-api-54d89bd5fc-s5g46\" (UID: \"b44c03e3-cd6a-41ee-9233-777271b72b2c\") " pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" Sep 11 15:14:36.955115 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:14:36.955065 2558 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 11 15:14:36.955154 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:14:36.955129 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b44c03e3-cd6a-41ee-9233-777271b72b2c-tls-certs podName:b44c03e3-cd6a-41ee-9233-777271b72b2c nodeName:}" failed. No retries permitted until 2026-09-11 15:14:37.455112831 +0000 UTC m=+371.687037259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b44c03e3-cd6a-41ee-9233-777271b72b2c-tls-certs") pod "model-serving-api-54d89bd5fc-s5g46" (UID: "b44c03e3-cd6a-41ee-9233-777271b72b2c") : secret "model-serving-api-tls" not found Sep 11 15:14:36.963393 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:36.963368 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjbn2\" (UniqueName: \"kubernetes.io/projected/b44c03e3-cd6a-41ee-9233-777271b72b2c-kube-api-access-jjbn2\") pod \"model-serving-api-54d89bd5fc-s5g46\" (UID: \"b44c03e3-cd6a-41ee-9233-777271b72b2c\") " pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" Sep 11 15:14:37.156104 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:37.156075 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ecd26b7-a4e6-4ae1-8485-3a619d964390-cert\") pod \"odh-model-controller-7998c67dc4-62ngs\" (UID: \"2ecd26b7-a4e6-4ae1-8485-3a619d964390\") " pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" Sep 11 15:14:37.158592 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:37.158566 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ecd26b7-a4e6-4ae1-8485-3a619d964390-cert\") pod \"odh-model-controller-7998c67dc4-62ngs\" (UID: \"2ecd26b7-a4e6-4ae1-8485-3a619d964390\") " pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" Sep 11 15:14:37.370993 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:37.370954 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" Sep 11 15:14:37.458646 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:37.458615 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44c03e3-cd6a-41ee-9233-777271b72b2c-tls-certs\") pod \"model-serving-api-54d89bd5fc-s5g46\" (UID: \"b44c03e3-cd6a-41ee-9233-777271b72b2c\") " pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" Sep 11 15:14:37.460921 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:37.460894 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b44c03e3-cd6a-41ee-9233-777271b72b2c-tls-certs\") pod \"model-serving-api-54d89bd5fc-s5g46\" (UID: \"b44c03e3-cd6a-41ee-9233-777271b72b2c\") " pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" Sep 11 15:14:37.502374 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:37.502345 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:37.642651 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:37.642609 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-62ngs"] Sep 11 15:14:37.644865 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:14:37.644842 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecd26b7_a4e6_4ae1_8485_3a619d964390.slice/crio-99a4ab751a6a57af10a061e0b0114b381ecc543269c9955fa2e769ac4b37d4d7 WatchSource:0}: Error finding container 99a4ab751a6a57af10a061e0b0114b381ecc543269c9955fa2e769ac4b37d4d7: Status 404 returned error can't find the container with id 99a4ab751a6a57af10a061e0b0114b381ecc543269c9955fa2e769ac4b37d4d7 Sep 11 15:14:37.686644 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:37.686586 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" Sep 11 15:14:37.850081 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:37.850049 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" event={"ID":"2ecd26b7-a4e6-4ae1-8485-3a619d964390","Type":"ContainerStarted","Data":"99a4ab751a6a57af10a061e0b0114b381ecc543269c9955fa2e769ac4b37d4d7"} Sep 11 15:14:37.959781 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:37.959752 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-s5g46"] Sep 11 15:14:37.961668 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:14:37.961645 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44c03e3_cd6a_41ee_9233_777271b72b2c.slice/crio-ef7f6605986d601685cc70dd6e06c303be9ecb72f144a3cefc20ac6785718b32 WatchSource:0}: Error finding container ef7f6605986d601685cc70dd6e06c303be9ecb72f144a3cefc20ac6785718b32: Status 404 returned error can't find the container with id ef7f6605986d601685cc70dd6e06c303be9ecb72f144a3cefc20ac6785718b32 Sep 11 15:14:38.857019 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:38.856978 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" event={"ID":"b44c03e3-cd6a-41ee-9233-777271b72b2c","Type":"ContainerStarted","Data":"ef7f6605986d601685cc70dd6e06c303be9ecb72f144a3cefc20ac6785718b32"} Sep 11 15:14:40.869805 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:40.869675 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" event={"ID":"2ecd26b7-a4e6-4ae1-8485-3a619d964390","Type":"ContainerStarted","Data":"01d169fc7e9ba54ee61a1697c5de5e3df68fbba3fb0b1016881ad85c9b881f8f"} Sep 11 15:14:40.870248 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:40.869852 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" Sep 11 15:14:40.887758 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:40.887695 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" podStartSLOduration=1.9981482000000002 podStartE2EDuration="4.887674401s" totalImagesPullingTime="2.889526201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:14:40.885715138 +0000 UTC m=+375.117639570" watchObservedRunningTime="2026-09-11 15:14:40.887674401 +0000 UTC m=+375.119598833" Sep 11 15:14:42.882379 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:42.882292 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" event={"ID":"b44c03e3-cd6a-41ee-9233-777271b72b2c","Type":"ContainerStarted","Data":"111c5052800c14057401e81399268f5d0cf7a19ecb639c78da8b465e2423ea61"} Sep 11 15:14:42.882379 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:42.882359 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" Sep 11 15:14:43.181679 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.181623 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" podStartSLOduration=2.640893025 podStartE2EDuration="7.181607212s" totalImagesPullingTime="4.540714187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:14:42.904266156 +0000 UTC m=+377.136190586" watchObservedRunningTime="2026-09-11 15:14:43.181607212 +0000 UTC m=+377.413531643" Sep 11 15:14:43.181865 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.181766 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8"] Sep 11 15:14:43.185089 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.185073 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.187656 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.187634 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gr6n7\"" Sep 11 15:14:43.187796 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.187676 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 15:14:43.187796 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.187680 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 15:14:43.187796 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.187686 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 15:14:43.188042 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.188020 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 15:14:43.188137 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.188047 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 15:14:43.188137 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.188055 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 15:14:43.199620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.199599 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8"] Sep 11 15:14:43.310096 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.310056 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.310096 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.310102 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.310336 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.310196 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhd65\" (UniqueName: \"kubernetes.io/projected/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-kube-api-access-jhd65\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.310336 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.310236 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.310336 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.310260 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.310481 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.310338 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.310481 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.310363 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.411138 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.411106 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.411138 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.411137 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.411320 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.411193 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.411320 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.411223 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.411320 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.411285 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhd65\" (UniqueName: \"kubernetes.io/projected/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-kube-api-access-jhd65\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.411320 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.411311 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.411522 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.411335 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.412033 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.412007 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.413495 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.413470 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.413661 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.413645 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.413706 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.413671 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.413804 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.413786 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.427546 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.427521 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.428023 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.428001 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhd65\" (UniqueName: \"kubernetes.io/projected/c9ba6cbd-6e02-4ef4-aeb7-989bb863027d-kube-api-access-jhd65\") pod \"istiod-openshift-gateway-5677ccb49b-4slk8\" (UID: \"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.494041 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.493989 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:43.770720 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.770640 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8"] Sep 11 15:14:43.773637 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:14:43.773609 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ba6cbd_6e02_4ef4_aeb7_989bb863027d.slice/crio-66449db14a8ba5c3586f227273eb3c3d5c4f1bf177fcacc6aa897a4b2015f25d WatchSource:0}: Error finding container 66449db14a8ba5c3586f227273eb3c3d5c4f1bf177fcacc6aa897a4b2015f25d: Status 404 returned error can't find the container with id 66449db14a8ba5c3586f227273eb3c3d5c4f1bf177fcacc6aa897a4b2015f25d Sep 11 15:14:43.888728 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:43.888688 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" event={"ID":"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d","Type":"ContainerStarted","Data":"66449db14a8ba5c3586f227273eb3c3d5c4f1bf177fcacc6aa897a4b2015f25d"} Sep 11 15:14:44.096536 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:44.096452 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:44.198286 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:44.198253 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:44.297696 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:44.297667 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:45.170188 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:45.170158 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:45.266951 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:45.266916 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:46.087376 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:46.087335 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:14:46.087464 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:46.087407 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:14:46.238477 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:46.238432 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:46.339674 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:46.339573 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:46.455877 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:46.455838 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:46.597846 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:46.597759 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:46.728498 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:46.728446 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:46.908670 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:46.908633 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" event={"ID":"c9ba6cbd-6e02-4ef4-aeb7-989bb863027d","Type":"ContainerStarted","Data":"3d991ebfe513f2296e76444a5dfb1f864ef0bdb8d23d123157a4c47fbcc79654"} Sep 11 15:14:46.908864 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:46.908777 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:46.971415 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:46.971359 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" podStartSLOduration=1.660397863 podStartE2EDuration="3.971343967s" totalImagesPullingTime="2.310946104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:14:46.970576738 +0000 UTC m=+381.202501169" watchObservedRunningTime="2026-09-11 15:14:46.971343967 +0000 UTC m=+381.203268397" Sep 11 15:14:47.686881 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:47.686844 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:47.776632 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:47.776597 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:47.877447 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:47.877413 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:14:47.914877 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:47.914852 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4slk8" Sep 11 15:14:51.879902 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:51.879868 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-62ngs" Sep 11 15:14:53.893231 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:14:53.893204 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-s5g46" Sep 11 15:15:05.808139 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:05.808035 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:05.866362 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:05.866331 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:05.944356 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:05.944329 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:06.547954 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:06.547918 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:06.697262 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:06.697228 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:16.317173 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:16.317134 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dc79d45f7-lng6r"] Sep 11 15:15:16.665365 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:16.665332 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:16.870436 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:16.870401 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:37.734432 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:37.734380 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:37.841545 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:37.841510 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:41.342382 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.342319 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6dc79d45f7-lng6r" podUID="6a0f4757-4dc5-4d5a-a938-ed26766e8e0e" containerName="console" containerID="cri-o://8bee888840dc80d68d1d874171eaba9c1e82bd76113c8f32d34c1fcb3b8df5ae" gracePeriod=15 Sep 11 15:15:41.581874 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.581846 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:15:41.685360 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.685334 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9z8j6\" (UniqueName: \"kubernetes.io/projected/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-kube-api-access-9z8j6\") pod \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " Sep 11 15:15:41.685541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.685370 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-oauth-config\") pod \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " Sep 11 15:15:41.685541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.685405 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-config\") pod \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " Sep 11 15:15:41.685541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.685421 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-service-ca\") pod \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " Sep 11 15:15:41.685541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.685442 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-oauth-serving-cert\") pod \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " Sep 11 15:15:41.685541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.685503 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-serving-cert\") pod \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " Sep 11 15:15:41.685541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.685533 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-trusted-ca-bundle\") pod \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\" (UID: \"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e\") " Sep 11 15:15:41.685908 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.685878 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-config" (OuterVolumeSpecName: "console-config") pod "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e" (UID: "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:15:41.685908 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.685892 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e" (UID: "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:15:41.686012 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.685888 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-service-ca" (OuterVolumeSpecName: "service-ca") pod "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e" (UID: "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:15:41.686143 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.686123 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e" (UID: "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:15:41.687625 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.687594 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e" (UID: "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:15:41.687726 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.687628 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-kube-api-access-9z8j6" (OuterVolumeSpecName: "kube-api-access-9z8j6") pod "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e" (UID: "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e"). InnerVolumeSpecName "kube-api-access-9z8j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:15:41.687726 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.687650 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e" (UID: "6a0f4757-4dc5-4d5a-a938-ed26766e8e0e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:15:41.786466 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.786431 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z8j6\" (UniqueName: \"kubernetes.io/projected/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-kube-api-access-9z8j6\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:41.786466 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.786463 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-oauth-config\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:41.786466 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.786478 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-config\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:41.786650 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.786488 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:41.786650 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.786496 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-oauth-serving-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:41.786650 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.786504 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-console-serving-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:41.786650 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.786513 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e-trusted-ca-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:41.968653 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.968573 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts"] Sep 11 15:15:41.968960 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.968945 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0f4757-4dc5-4d5a-a938-ed26766e8e0e" containerName="console" Sep 11 15:15:41.969009 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.968963 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0f4757-4dc5-4d5a-a938-ed26766e8e0e" containerName="console" Sep 11 15:15:41.969043 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.969020 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0f4757-4dc5-4d5a-a938-ed26766e8e0e" containerName="console" Sep 11 15:15:41.972570 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.972556 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" Sep 11 15:15:41.974992 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.974969 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 15:15:41.975089 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.974999 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 15:15:41.975089 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.975011 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sztdq\"" Sep 11 15:15:41.983958 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:41.983279 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts"] Sep 11 15:15:42.060461 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.060430 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz"] Sep 11 15:15:42.064037 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.064021 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" Sep 11 15:15:42.073013 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.072988 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz"] Sep 11 15:15:42.089064 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.089035 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25z85\" (UniqueName: \"kubernetes.io/projected/921ef911-dbdd-4b2d-8d73-73779c53c166-kube-api-access-25z85\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts\" (UID: \"921ef911-dbdd-4b2d-8d73-73779c53c166\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" Sep 11 15:15:42.089176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.089115 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/921ef911-dbdd-4b2d-8d73-73779c53c166-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts\" (UID: \"921ef911-dbdd-4b2d-8d73-73779c53c166\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" Sep 11 15:15:42.089176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.089155 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/921ef911-dbdd-4b2d-8d73-73779c53c166-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts\" (UID: \"921ef911-dbdd-4b2d-8d73-73779c53c166\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" Sep 11 15:15:42.164489 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.164457 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l"] Sep 11 15:15:42.166090 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.166066 2558 generic.go:356] "Generic (PLEG): container finished" podID="6a0f4757-4dc5-4d5a-a938-ed26766e8e0e" containerID="8bee888840dc80d68d1d874171eaba9c1e82bd76113c8f32d34c1fcb3b8df5ae" exitCode=0 Sep 11 15:15:42.166192 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.166177 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dc79d45f7-lng6r" Sep 11 15:15:42.168327 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.168305 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc79d45f7-lng6r" event={"ID":"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e","Type":"ContainerDied","Data":"8bee888840dc80d68d1d874171eaba9c1e82bd76113c8f32d34c1fcb3b8df5ae"} Sep 11 15:15:42.168410 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.168337 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dc79d45f7-lng6r" event={"ID":"6a0f4757-4dc5-4d5a-a938-ed26766e8e0e","Type":"ContainerDied","Data":"f2c1fc30e25c6a6c53800f9280ab87db0d4d997b01bfdb64676d83daf600a788"} Sep 11 15:15:42.168410 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.168363 2558 scope.go:117] "RemoveContainer" containerID="8bee888840dc80d68d1d874171eaba9c1e82bd76113c8f32d34c1fcb3b8df5ae" Sep 11 15:15:42.168480 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.168420 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" Sep 11 15:15:42.178084 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.177947 2558 scope.go:117] "RemoveContainer" containerID="8bee888840dc80d68d1d874171eaba9c1e82bd76113c8f32d34c1fcb3b8df5ae" Sep 11 15:15:42.178363 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:15:42.178340 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bee888840dc80d68d1d874171eaba9c1e82bd76113c8f32d34c1fcb3b8df5ae\": container with ID starting with 8bee888840dc80d68d1d874171eaba9c1e82bd76113c8f32d34c1fcb3b8df5ae not found: ID does not exist" containerID="8bee888840dc80d68d1d874171eaba9c1e82bd76113c8f32d34c1fcb3b8df5ae" Sep 11 15:15:42.178439 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.178377 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bee888840dc80d68d1d874171eaba9c1e82bd76113c8f32d34c1fcb3b8df5ae"} err="failed to get container status \"8bee888840dc80d68d1d874171eaba9c1e82bd76113c8f32d34c1fcb3b8df5ae\": rpc error: code = NotFound desc = could not find container \"8bee888840dc80d68d1d874171eaba9c1e82bd76113c8f32d34c1fcb3b8df5ae\": container with ID starting with 8bee888840dc80d68d1d874171eaba9c1e82bd76113c8f32d34c1fcb3b8df5ae not found: ID does not exist" Sep 11 15:15:42.179238 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.179219 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l"] Sep 11 15:15:42.189854 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.189829 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b1e1ea-e87e-42e8-b01f-7b721f102073-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz\" (UID: \"66b1e1ea-e87e-42e8-b01f-7b721f102073\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" Sep 11 15:15:42.189944 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.189883 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/921ef911-dbdd-4b2d-8d73-73779c53c166-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts\" (UID: \"921ef911-dbdd-4b2d-8d73-73779c53c166\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" Sep 11 15:15:42.189944 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.189905 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/921ef911-dbdd-4b2d-8d73-73779c53c166-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts\" (UID: \"921ef911-dbdd-4b2d-8d73-73779c53c166\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" Sep 11 15:15:42.189944 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.189923 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66b1e1ea-e87e-42e8-b01f-7b721f102073-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz\" (UID: \"66b1e1ea-e87e-42e8-b01f-7b721f102073\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" Sep 11 15:15:42.189944 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.189942 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzj64\" (UniqueName: \"kubernetes.io/projected/66b1e1ea-e87e-42e8-b01f-7b721f102073-kube-api-access-wzj64\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz\" (UID: \"66b1e1ea-e87e-42e8-b01f-7b721f102073\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" Sep 11 15:15:42.190132 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.189967 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25z85\" (UniqueName: \"kubernetes.io/projected/921ef911-dbdd-4b2d-8d73-73779c53c166-kube-api-access-25z85\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts\" (UID: \"921ef911-dbdd-4b2d-8d73-73779c53c166\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" Sep 11 15:15:42.190198 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.190180 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/921ef911-dbdd-4b2d-8d73-73779c53c166-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts\" (UID: \"921ef911-dbdd-4b2d-8d73-73779c53c166\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" Sep 11 15:15:42.190265 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.190241 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/921ef911-dbdd-4b2d-8d73-73779c53c166-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts\" (UID: \"921ef911-dbdd-4b2d-8d73-73779c53c166\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" Sep 11 15:15:42.198217 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.198195 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25z85\" (UniqueName: \"kubernetes.io/projected/921ef911-dbdd-4b2d-8d73-73779c53c166-kube-api-access-25z85\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts\" (UID: \"921ef911-dbdd-4b2d-8d73-73779c53c166\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" Sep 11 15:15:42.203971 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.203947 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dc79d45f7-lng6r"] Sep 11 15:15:42.208342 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.208322 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dc79d45f7-lng6r"] Sep 11 15:15:42.264762 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.264690 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a0f4757-4dc5-4d5a-a938-ed26766e8e0e" path="/var/lib/kubelet/pods/6a0f4757-4dc5-4d5a-a938-ed26766e8e0e/volumes" Sep 11 15:15:42.267428 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.267410 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22"] Sep 11 15:15:42.272427 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.272413 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" Sep 11 15:15:42.278690 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.278669 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22"] Sep 11 15:15:42.287531 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.287511 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" Sep 11 15:15:42.290427 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.290304 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/492f19dd-556c-48eb-9573-74d47069bb63-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l\" (UID: \"492f19dd-556c-48eb-9573-74d47069bb63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" Sep 11 15:15:42.290427 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.290348 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/492f19dd-556c-48eb-9573-74d47069bb63-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l\" (UID: \"492f19dd-556c-48eb-9573-74d47069bb63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" Sep 11 15:15:42.290427 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.290407 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66b1e1ea-e87e-42e8-b01f-7b721f102073-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz\" (UID: \"66b1e1ea-e87e-42e8-b01f-7b721f102073\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" Sep 11 15:15:42.290659 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.290429 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzj64\" (UniqueName: \"kubernetes.io/projected/66b1e1ea-e87e-42e8-b01f-7b721f102073-kube-api-access-wzj64\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz\" (UID: \"66b1e1ea-e87e-42e8-b01f-7b721f102073\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" Sep 11 15:15:42.290659 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.290457 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzkwt\" (UniqueName: \"kubernetes.io/projected/492f19dd-556c-48eb-9573-74d47069bb63-kube-api-access-qzkwt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l\" (UID: \"492f19dd-556c-48eb-9573-74d47069bb63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" Sep 11 15:15:42.290659 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.290512 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b1e1ea-e87e-42e8-b01f-7b721f102073-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz\" (UID: \"66b1e1ea-e87e-42e8-b01f-7b721f102073\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" Sep 11 15:15:42.290902 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.290866 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66b1e1ea-e87e-42e8-b01f-7b721f102073-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz\" (UID: \"66b1e1ea-e87e-42e8-b01f-7b721f102073\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" Sep 11 15:15:42.290902 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.290888 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b1e1ea-e87e-42e8-b01f-7b721f102073-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz\" (UID: \"66b1e1ea-e87e-42e8-b01f-7b721f102073\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" Sep 11 15:15:42.298245 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.298225 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzj64\" (UniqueName: \"kubernetes.io/projected/66b1e1ea-e87e-42e8-b01f-7b721f102073-kube-api-access-wzj64\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz\" (UID: \"66b1e1ea-e87e-42e8-b01f-7b721f102073\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" Sep 11 15:15:42.373870 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.373836 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" Sep 11 15:15:42.391531 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.391502 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/693550bb-21d6-4baa-a395-eec7327cd99e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22\" (UID: \"693550bb-21d6-4baa-a395-eec7327cd99e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" Sep 11 15:15:42.391634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.391545 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/492f19dd-556c-48eb-9573-74d47069bb63-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l\" (UID: \"492f19dd-556c-48eb-9573-74d47069bb63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" Sep 11 15:15:42.391634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.391577 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/492f19dd-556c-48eb-9573-74d47069bb63-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l\" (UID: \"492f19dd-556c-48eb-9573-74d47069bb63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" Sep 11 15:15:42.391634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.391628 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzkwt\" (UniqueName: \"kubernetes.io/projected/492f19dd-556c-48eb-9573-74d47069bb63-kube-api-access-qzkwt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l\" (UID: \"492f19dd-556c-48eb-9573-74d47069bb63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" Sep 11 15:15:42.391763 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.391661 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mckrj\" (UniqueName: \"kubernetes.io/projected/693550bb-21d6-4baa-a395-eec7327cd99e-kube-api-access-mckrj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22\" (UID: \"693550bb-21d6-4baa-a395-eec7327cd99e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" Sep 11 15:15:42.391763 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.391697 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/693550bb-21d6-4baa-a395-eec7327cd99e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22\" (UID: \"693550bb-21d6-4baa-a395-eec7327cd99e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" Sep 11 15:15:42.391928 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.391909 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/492f19dd-556c-48eb-9573-74d47069bb63-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l\" (UID: \"492f19dd-556c-48eb-9573-74d47069bb63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" Sep 11 15:15:42.392004 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.391980 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/492f19dd-556c-48eb-9573-74d47069bb63-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l\" (UID: \"492f19dd-556c-48eb-9573-74d47069bb63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" Sep 11 15:15:42.401299 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.401273 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzkwt\" (UniqueName: \"kubernetes.io/projected/492f19dd-556c-48eb-9573-74d47069bb63-kube-api-access-qzkwt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l\" (UID: \"492f19dd-556c-48eb-9573-74d47069bb63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" Sep 11 15:15:42.481197 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.481165 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" Sep 11 15:15:42.492246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.492218 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mckrj\" (UniqueName: \"kubernetes.io/projected/693550bb-21d6-4baa-a395-eec7327cd99e-kube-api-access-mckrj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22\" (UID: \"693550bb-21d6-4baa-a395-eec7327cd99e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" Sep 11 15:15:42.492326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.492273 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/693550bb-21d6-4baa-a395-eec7327cd99e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22\" (UID: \"693550bb-21d6-4baa-a395-eec7327cd99e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" Sep 11 15:15:42.492371 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.492341 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/693550bb-21d6-4baa-a395-eec7327cd99e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22\" (UID: \"693550bb-21d6-4baa-a395-eec7327cd99e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" Sep 11 15:15:42.492649 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.492629 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/693550bb-21d6-4baa-a395-eec7327cd99e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22\" (UID: \"693550bb-21d6-4baa-a395-eec7327cd99e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" Sep 11 15:15:42.492721 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.492701 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/693550bb-21d6-4baa-a395-eec7327cd99e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22\" (UID: \"693550bb-21d6-4baa-a395-eec7327cd99e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" Sep 11 15:15:42.501512 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.501489 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mckrj\" (UniqueName: \"kubernetes.io/projected/693550bb-21d6-4baa-a395-eec7327cd99e-kube-api-access-mckrj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22\" (UID: \"693550bb-21d6-4baa-a395-eec7327cd99e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" Sep 11 15:15:42.561890 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.561862 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts"] Sep 11 15:15:42.562970 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:15:42.562942 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921ef911_dbdd_4b2d_8d73_73779c53c166.slice/crio-3a1352c798b98683ae202c0bfbe9993467ecd6f2ff74fbc96308464cf452141e WatchSource:0}: Error finding container 3a1352c798b98683ae202c0bfbe9993467ecd6f2ff74fbc96308464cf452141e: Status 404 returned error can't find the container with id 3a1352c798b98683ae202c0bfbe9993467ecd6f2ff74fbc96308464cf452141e Sep 11 15:15:42.582293 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.582271 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" Sep 11 15:15:42.648242 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.648216 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz"] Sep 11 15:15:42.650434 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:15:42.650401 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b1e1ea_e87e_42e8_b01f_7b721f102073.slice/crio-36a4fd4e6879208607a6a0bb749ea895dfe2d4c9689b0b0601523e7f2c9caa12 WatchSource:0}: Error finding container 36a4fd4e6879208607a6a0bb749ea895dfe2d4c9689b0b0601523e7f2c9caa12: Status 404 returned error can't find the container with id 36a4fd4e6879208607a6a0bb749ea895dfe2d4c9689b0b0601523e7f2c9caa12 Sep 11 15:15:42.750727 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.750703 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l"] Sep 11 15:15:42.752267 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:15:42.752245 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492f19dd_556c_48eb_9573_74d47069bb63.slice/crio-2753bca6c59f7971ee71a746e6daf24b820fc6ec4d8d7c230a2c9a7b4ca6eb04 WatchSource:0}: Error finding container 2753bca6c59f7971ee71a746e6daf24b820fc6ec4d8d7c230a2c9a7b4ca6eb04: Status 404 returned error can't find the container with id 2753bca6c59f7971ee71a746e6daf24b820fc6ec4d8d7c230a2c9a7b4ca6eb04 Sep 11 15:15:42.856802 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:42.856771 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22"] Sep 11 15:15:42.907544 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:15:42.907513 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693550bb_21d6_4baa_a395_eec7327cd99e.slice/crio-d33d6f8ea368596255b07b044b81be28c71c0a71ecff303c287150624ecdee90 WatchSource:0}: Error finding container d33d6f8ea368596255b07b044b81be28c71c0a71ecff303c287150624ecdee90: Status 404 returned error can't find the container with id d33d6f8ea368596255b07b044b81be28c71c0a71ecff303c287150624ecdee90 Sep 11 15:15:43.172875 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:43.172844 2558 generic.go:356] "Generic (PLEG): container finished" podID="492f19dd-556c-48eb-9573-74d47069bb63" containerID="b4d4a7873c1a14d8cd052d9bf41dace3a6dc6c965f28ffe7ebe6ffb18ea3cd61" exitCode=0 Sep 11 15:15:43.173050 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:43.172929 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" event={"ID":"492f19dd-556c-48eb-9573-74d47069bb63","Type":"ContainerDied","Data":"b4d4a7873c1a14d8cd052d9bf41dace3a6dc6c965f28ffe7ebe6ffb18ea3cd61"} Sep 11 15:15:43.173050 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:43.172989 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" event={"ID":"492f19dd-556c-48eb-9573-74d47069bb63","Type":"ContainerStarted","Data":"2753bca6c59f7971ee71a746e6daf24b820fc6ec4d8d7c230a2c9a7b4ca6eb04"} Sep 11 15:15:43.174555 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:43.174536 2558 generic.go:356] "Generic (PLEG): container finished" podID="66b1e1ea-e87e-42e8-b01f-7b721f102073" containerID="0ca6b05d26bc220efa46144864b2eab07d0b1e2044c8e9fd0dd7901c601396d1" exitCode=0 Sep 11 15:15:43.174665 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:43.174620 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" event={"ID":"66b1e1ea-e87e-42e8-b01f-7b721f102073","Type":"ContainerDied","Data":"0ca6b05d26bc220efa46144864b2eab07d0b1e2044c8e9fd0dd7901c601396d1"} Sep 11 15:15:43.174665 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:43.174661 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" event={"ID":"66b1e1ea-e87e-42e8-b01f-7b721f102073","Type":"ContainerStarted","Data":"36a4fd4e6879208607a6a0bb749ea895dfe2d4c9689b0b0601523e7f2c9caa12"} Sep 11 15:15:43.176218 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:43.176192 2558 generic.go:356] "Generic (PLEG): container finished" podID="693550bb-21d6-4baa-a395-eec7327cd99e" containerID="cde701f5832a24beb56181c40b4d56e2496e46b5f897cdb67a49aa570dceca26" exitCode=0 Sep 11 15:15:43.176218 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:43.176222 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" event={"ID":"693550bb-21d6-4baa-a395-eec7327cd99e","Type":"ContainerDied","Data":"cde701f5832a24beb56181c40b4d56e2496e46b5f897cdb67a49aa570dceca26"} Sep 11 15:15:43.176386 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:43.176262 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" event={"ID":"693550bb-21d6-4baa-a395-eec7327cd99e","Type":"ContainerStarted","Data":"d33d6f8ea368596255b07b044b81be28c71c0a71ecff303c287150624ecdee90"} Sep 11 15:15:43.178126 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:43.178105 2558 generic.go:356] "Generic (PLEG): container finished" podID="921ef911-dbdd-4b2d-8d73-73779c53c166" containerID="1b7875f9063471836cf2f18588be822749b8df3518b83bf4bf53f8bf610ae673" exitCode=0 Sep 11 15:15:43.178231 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:43.178183 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" event={"ID":"921ef911-dbdd-4b2d-8d73-73779c53c166","Type":"ContainerDied","Data":"1b7875f9063471836cf2f18588be822749b8df3518b83bf4bf53f8bf610ae673"} Sep 11 15:15:43.178231 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:43.178218 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" event={"ID":"921ef911-dbdd-4b2d-8d73-73779c53c166","Type":"ContainerStarted","Data":"3a1352c798b98683ae202c0bfbe9993467ecd6f2ff74fbc96308464cf452141e"} Sep 11 15:15:44.186785 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:44.186693 2558 generic.go:356] "Generic (PLEG): container finished" podID="693550bb-21d6-4baa-a395-eec7327cd99e" containerID="3fad4da8dee1734b49b42f0dc3740a0a1e2b8b3e3adbe3eb1fe148a2122832b9" exitCode=0 Sep 11 15:15:44.187234 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:44.186782 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" event={"ID":"693550bb-21d6-4baa-a395-eec7327cd99e","Type":"ContainerDied","Data":"3fad4da8dee1734b49b42f0dc3740a0a1e2b8b3e3adbe3eb1fe148a2122832b9"} Sep 11 15:15:45.197571 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:45.197542 2558 generic.go:356] "Generic (PLEG): container finished" podID="693550bb-21d6-4baa-a395-eec7327cd99e" containerID="503da539125ba2948385192227d30c1998a8fc3c0f2e6cc49b415ccfe7a87e96" exitCode=0 Sep 11 15:15:45.197983 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:45.197616 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" event={"ID":"693550bb-21d6-4baa-a395-eec7327cd99e","Type":"ContainerDied","Data":"503da539125ba2948385192227d30c1998a8fc3c0f2e6cc49b415ccfe7a87e96"} Sep 11 15:15:45.199426 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:45.199402 2558 generic.go:356] "Generic (PLEG): container finished" podID="921ef911-dbdd-4b2d-8d73-73779c53c166" containerID="a18f97f09a66bf0cd3ef2b58dfde295c8da5777a2d921eb0480475cff38e84bd" exitCode=0 Sep 11 15:15:45.199524 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:45.199476 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" event={"ID":"921ef911-dbdd-4b2d-8d73-73779c53c166","Type":"ContainerDied","Data":"a18f97f09a66bf0cd3ef2b58dfde295c8da5777a2d921eb0480475cff38e84bd"} Sep 11 15:15:45.201312 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:45.201292 2558 generic.go:356] "Generic (PLEG): container finished" podID="492f19dd-556c-48eb-9573-74d47069bb63" containerID="838193cefa94cc2a477397363fca7afa2eeb7012579dd9e5977b03a632af6848" exitCode=0 Sep 11 15:15:45.201412 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:45.201336 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" event={"ID":"492f19dd-556c-48eb-9573-74d47069bb63","Type":"ContainerDied","Data":"838193cefa94cc2a477397363fca7afa2eeb7012579dd9e5977b03a632af6848"} Sep 11 15:15:45.203341 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:45.203321 2558 generic.go:356] "Generic (PLEG): container finished" podID="66b1e1ea-e87e-42e8-b01f-7b721f102073" containerID="87cbfb459435212c4b925b6aeaece7b9af19a1958b2b776ec3cd4c8083809cb1" exitCode=0 Sep 11 15:15:45.203442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:45.203352 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" event={"ID":"66b1e1ea-e87e-42e8-b01f-7b721f102073","Type":"ContainerDied","Data":"87cbfb459435212c4b925b6aeaece7b9af19a1958b2b776ec3cd4c8083809cb1"} Sep 11 15:15:46.210544 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.210515 2558 generic.go:356] "Generic (PLEG): container finished" podID="921ef911-dbdd-4b2d-8d73-73779c53c166" containerID="59bb488d3eab43e1da769d5eab3aaf1482f7a4eebf84302927b1c40345342e5f" exitCode=0 Sep 11 15:15:46.210937 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.210625 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" event={"ID":"921ef911-dbdd-4b2d-8d73-73779c53c166","Type":"ContainerDied","Data":"59bb488d3eab43e1da769d5eab3aaf1482f7a4eebf84302927b1c40345342e5f"} Sep 11 15:15:46.212648 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.212627 2558 generic.go:356] "Generic (PLEG): container finished" podID="492f19dd-556c-48eb-9573-74d47069bb63" containerID="9e77d0c011046095d372cce2dc3263980544a898a5978745ec4639caf12068d5" exitCode=0 Sep 11 15:15:46.212757 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.212657 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" event={"ID":"492f19dd-556c-48eb-9573-74d47069bb63","Type":"ContainerDied","Data":"9e77d0c011046095d372cce2dc3263980544a898a5978745ec4639caf12068d5"} Sep 11 15:15:46.214802 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.214778 2558 generic.go:356] "Generic (PLEG): container finished" podID="66b1e1ea-e87e-42e8-b01f-7b721f102073" containerID="4d34636cc05c6869d6bf90923478012e6952d4ad414261ce9ecee60c0d890c86" exitCode=0 Sep 11 15:15:46.214922 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.214804 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" event={"ID":"66b1e1ea-e87e-42e8-b01f-7b721f102073","Type":"ContainerDied","Data":"4d34636cc05c6869d6bf90923478012e6952d4ad414261ce9ecee60c0d890c86"} Sep 11 15:15:46.343911 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.343886 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" Sep 11 15:15:46.427684 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.427651 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/693550bb-21d6-4baa-a395-eec7327cd99e-util\") pod \"693550bb-21d6-4baa-a395-eec7327cd99e\" (UID: \"693550bb-21d6-4baa-a395-eec7327cd99e\") " Sep 11 15:15:46.427862 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.427720 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mckrj\" (UniqueName: \"kubernetes.io/projected/693550bb-21d6-4baa-a395-eec7327cd99e-kube-api-access-mckrj\") pod \"693550bb-21d6-4baa-a395-eec7327cd99e\" (UID: \"693550bb-21d6-4baa-a395-eec7327cd99e\") " Sep 11 15:15:46.427862 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.427759 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/693550bb-21d6-4baa-a395-eec7327cd99e-bundle\") pod \"693550bb-21d6-4baa-a395-eec7327cd99e\" (UID: \"693550bb-21d6-4baa-a395-eec7327cd99e\") " Sep 11 15:15:46.428276 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.428251 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693550bb-21d6-4baa-a395-eec7327cd99e-bundle" (OuterVolumeSpecName: "bundle") pod "693550bb-21d6-4baa-a395-eec7327cd99e" (UID: "693550bb-21d6-4baa-a395-eec7327cd99e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:15:46.429922 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.429898 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693550bb-21d6-4baa-a395-eec7327cd99e-kube-api-access-mckrj" (OuterVolumeSpecName: "kube-api-access-mckrj") pod "693550bb-21d6-4baa-a395-eec7327cd99e" (UID: "693550bb-21d6-4baa-a395-eec7327cd99e"). InnerVolumeSpecName "kube-api-access-mckrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:15:46.435145 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.435112 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693550bb-21d6-4baa-a395-eec7327cd99e-util" (OuterVolumeSpecName: "util") pod "693550bb-21d6-4baa-a395-eec7327cd99e" (UID: "693550bb-21d6-4baa-a395-eec7327cd99e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:15:46.529026 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.528956 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/693550bb-21d6-4baa-a395-eec7327cd99e-util\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:46.529026 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.528982 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mckrj\" (UniqueName: \"kubernetes.io/projected/693550bb-21d6-4baa-a395-eec7327cd99e-kube-api-access-mckrj\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:46.529026 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:46.528993 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/693550bb-21d6-4baa-a395-eec7327cd99e-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:47.222129 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.222097 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" Sep 11 15:15:47.222129 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.222111 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6zcs22" event={"ID":"693550bb-21d6-4baa-a395-eec7327cd99e","Type":"ContainerDied","Data":"d33d6f8ea368596255b07b044b81be28c71c0a71ecff303c287150624ecdee90"} Sep 11 15:15:47.222624 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.222161 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d33d6f8ea368596255b07b044b81be28c71c0a71ecff303c287150624ecdee90" Sep 11 15:15:47.369686 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.369654 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" Sep 11 15:15:47.403773 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.403733 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" Sep 11 15:15:47.407958 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.407935 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" Sep 11 15:15:47.437639 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.437615 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b1e1ea-e87e-42e8-b01f-7b721f102073-bundle\") pod \"66b1e1ea-e87e-42e8-b01f-7b721f102073\" (UID: \"66b1e1ea-e87e-42e8-b01f-7b721f102073\") " Sep 11 15:15:47.437889 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.437722 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzj64\" (UniqueName: \"kubernetes.io/projected/66b1e1ea-e87e-42e8-b01f-7b721f102073-kube-api-access-wzj64\") pod \"66b1e1ea-e87e-42e8-b01f-7b721f102073\" (UID: \"66b1e1ea-e87e-42e8-b01f-7b721f102073\") " Sep 11 15:15:47.437889 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.437777 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66b1e1ea-e87e-42e8-b01f-7b721f102073-util\") pod \"66b1e1ea-e87e-42e8-b01f-7b721f102073\" (UID: \"66b1e1ea-e87e-42e8-b01f-7b721f102073\") " Sep 11 15:15:47.440299 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.438693 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b1e1ea-e87e-42e8-b01f-7b721f102073-bundle" (OuterVolumeSpecName: "bundle") pod "66b1e1ea-e87e-42e8-b01f-7b721f102073" (UID: "66b1e1ea-e87e-42e8-b01f-7b721f102073"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:15:47.441004 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.440913 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b1e1ea-e87e-42e8-b01f-7b721f102073-kube-api-access-wzj64" (OuterVolumeSpecName: "kube-api-access-wzj64") pod "66b1e1ea-e87e-42e8-b01f-7b721f102073" (UID: "66b1e1ea-e87e-42e8-b01f-7b721f102073"). InnerVolumeSpecName "kube-api-access-wzj64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:15:47.444992 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.444955 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b1e1ea-e87e-42e8-b01f-7b721f102073-util" (OuterVolumeSpecName: "util") pod "66b1e1ea-e87e-42e8-b01f-7b721f102073" (UID: "66b1e1ea-e87e-42e8-b01f-7b721f102073"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:15:47.539281 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.539196 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzkwt\" (UniqueName: \"kubernetes.io/projected/492f19dd-556c-48eb-9573-74d47069bb63-kube-api-access-qzkwt\") pod \"492f19dd-556c-48eb-9573-74d47069bb63\" (UID: \"492f19dd-556c-48eb-9573-74d47069bb63\") " Sep 11 15:15:47.539281 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.539248 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/921ef911-dbdd-4b2d-8d73-73779c53c166-bundle\") pod \"921ef911-dbdd-4b2d-8d73-73779c53c166\" (UID: \"921ef911-dbdd-4b2d-8d73-73779c53c166\") " Sep 11 15:15:47.539281 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.539272 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/492f19dd-556c-48eb-9573-74d47069bb63-util\") pod \"492f19dd-556c-48eb-9573-74d47069bb63\" (UID: \"492f19dd-556c-48eb-9573-74d47069bb63\") " Sep 11 15:15:47.539489 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.539358 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/492f19dd-556c-48eb-9573-74d47069bb63-bundle\") pod \"492f19dd-556c-48eb-9573-74d47069bb63\" (UID: \"492f19dd-556c-48eb-9573-74d47069bb63\") " Sep 11 15:15:47.539489 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.539405 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/921ef911-dbdd-4b2d-8d73-73779c53c166-util\") pod \"921ef911-dbdd-4b2d-8d73-73779c53c166\" (UID: \"921ef911-dbdd-4b2d-8d73-73779c53c166\") " Sep 11 15:15:47.539489 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.539453 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25z85\" (UniqueName: \"kubernetes.io/projected/921ef911-dbdd-4b2d-8d73-73779c53c166-kube-api-access-25z85\") pod \"921ef911-dbdd-4b2d-8d73-73779c53c166\" (UID: \"921ef911-dbdd-4b2d-8d73-73779c53c166\") " Sep 11 15:15:47.539794 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.539695 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzj64\" (UniqueName: \"kubernetes.io/projected/66b1e1ea-e87e-42e8-b01f-7b721f102073-kube-api-access-wzj64\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:47.539794 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.539733 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66b1e1ea-e87e-42e8-b01f-7b721f102073-util\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:47.539794 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.539775 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66b1e1ea-e87e-42e8-b01f-7b721f102073-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:47.540022 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.539990 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492f19dd-556c-48eb-9573-74d47069bb63-bundle" (OuterVolumeSpecName: "bundle") pod "492f19dd-556c-48eb-9573-74d47069bb63" (UID: "492f19dd-556c-48eb-9573-74d47069bb63"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:15:47.540160 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.540127 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921ef911-dbdd-4b2d-8d73-73779c53c166-bundle" (OuterVolumeSpecName: "bundle") pod "921ef911-dbdd-4b2d-8d73-73779c53c166" (UID: "921ef911-dbdd-4b2d-8d73-73779c53c166"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:15:47.541523 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.541501 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492f19dd-556c-48eb-9573-74d47069bb63-kube-api-access-qzkwt" (OuterVolumeSpecName: "kube-api-access-qzkwt") pod "492f19dd-556c-48eb-9573-74d47069bb63" (UID: "492f19dd-556c-48eb-9573-74d47069bb63"). InnerVolumeSpecName "kube-api-access-qzkwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:15:47.541620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.541538 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921ef911-dbdd-4b2d-8d73-73779c53c166-kube-api-access-25z85" (OuterVolumeSpecName: "kube-api-access-25z85") pod "921ef911-dbdd-4b2d-8d73-73779c53c166" (UID: "921ef911-dbdd-4b2d-8d73-73779c53c166"). InnerVolumeSpecName "kube-api-access-25z85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:15:47.545154 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.545121 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492f19dd-556c-48eb-9573-74d47069bb63-util" (OuterVolumeSpecName: "util") pod "492f19dd-556c-48eb-9573-74d47069bb63" (UID: "492f19dd-556c-48eb-9573-74d47069bb63"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:15:47.545543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.545519 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921ef911-dbdd-4b2d-8d73-73779c53c166-util" (OuterVolumeSpecName: "util") pod "921ef911-dbdd-4b2d-8d73-73779c53c166" (UID: "921ef911-dbdd-4b2d-8d73-73779c53c166"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:15:47.640496 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.640444 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/492f19dd-556c-48eb-9573-74d47069bb63-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:47.640496 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.640488 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/921ef911-dbdd-4b2d-8d73-73779c53c166-util\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:47.640496 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.640501 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25z85\" (UniqueName: \"kubernetes.io/projected/921ef911-dbdd-4b2d-8d73-73779c53c166-kube-api-access-25z85\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:47.640719 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.640514 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzkwt\" (UniqueName: \"kubernetes.io/projected/492f19dd-556c-48eb-9573-74d47069bb63-kube-api-access-qzkwt\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:47.640719 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.640528 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/921ef911-dbdd-4b2d-8d73-73779c53c166-bundle\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:47.640719 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:47.640539 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/492f19dd-556c-48eb-9573-74d47069bb63-util\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:15:48.228948 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:48.228921 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" Sep 11 15:15:48.229342 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:48.228916 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z9mbz" event={"ID":"66b1e1ea-e87e-42e8-b01f-7b721f102073","Type":"ContainerDied","Data":"36a4fd4e6879208607a6a0bb749ea895dfe2d4c9689b0b0601523e7f2c9caa12"} Sep 11 15:15:48.229342 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:48.229049 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36a4fd4e6879208607a6a0bb749ea895dfe2d4c9689b0b0601523e7f2c9caa12" Sep 11 15:15:48.230978 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:48.230957 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" Sep 11 15:15:48.230978 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:48.230967 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an9gts" event={"ID":"921ef911-dbdd-4b2d-8d73-73779c53c166","Type":"ContainerDied","Data":"3a1352c798b98683ae202c0bfbe9993467ecd6f2ff74fbc96308464cf452141e"} Sep 11 15:15:48.231176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:48.231009 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a1352c798b98683ae202c0bfbe9993467ecd6f2ff74fbc96308464cf452141e" Sep 11 15:15:48.233003 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:48.232983 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" event={"ID":"492f19dd-556c-48eb-9573-74d47069bb63","Type":"ContainerDied","Data":"2753bca6c59f7971ee71a746e6daf24b820fc6ec4d8d7c230a2c9a7b4ca6eb04"} Sep 11 15:15:48.233103 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:48.233015 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2753bca6c59f7971ee71a746e6daf24b820fc6ec4d8d7c230a2c9a7b4ca6eb04" Sep 11 15:15:48.233103 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:48.233041 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c88m2l" Sep 11 15:15:49.276386 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:49.276313 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:49.377172 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:49.377125 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:52.419453 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:52.419424 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:53.201363 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:53.201333 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:53.991940 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:53.991908 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:57.004409 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:57.004369 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:57.298543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:57.298437 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:57.409657 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:57.409625 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:57.511223 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:57.511195 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:59.029884 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:59.029852 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:15:59.313432 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:15:59.313350 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:00.029722 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:00.029690 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:00.447937 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:00.447908 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:02.236730 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:02.236693 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:03.701390 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:03.701352 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:03.982843 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:03.982768 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:04.125247 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:04.125216 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:07.648087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:07.648047 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:07.746309 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:07.746279 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:08.149497 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:08.149451 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:08.293157 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:08.293124 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:08.540430 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:08.540355 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:08.689415 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:08.689382 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:16.778043 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:16.778001 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:17.065647 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:17.065571 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:17.393176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:17.393145 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:19.444897 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:19.444866 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:19.630231 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:19.630196 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:23.915352 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:23.915315 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:24.179160 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:24.179080 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:29.130788 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:29.130731 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:29.315368 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:29.315330 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:29.480040 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:29.480006 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:37.834366 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:37.834328 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:37.931787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:37.931719 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:40.380205 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:40.380164 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:40.625504 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:40.625467 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:45.035575 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:45.035538 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:45.136163 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:45.136120 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:49.294560 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:49.294530 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:49.598619 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:49.598538 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:50.088652 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:50.088620 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:50.189129 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:50.189096 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:50.288927 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:50.288890 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:50.641518 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:50.641485 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:50.702705 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:50.702672 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:50.831550 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:50.831518 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:50.934870 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:50.934787 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:51.002787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.002752 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:51.105490 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.105451 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:51.170952 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.170911 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:51.361668 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.361589 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:16:51.361949 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.361934 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="921ef911-dbdd-4b2d-8d73-73779c53c166" containerName="pull" Sep 11 15:16:51.361949 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.361949 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="921ef911-dbdd-4b2d-8d73-73779c53c166" containerName="pull" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.361960 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="921ef911-dbdd-4b2d-8d73-73779c53c166" containerName="extract" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.361967 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="921ef911-dbdd-4b2d-8d73-73779c53c166" containerName="extract" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.361975 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693550bb-21d6-4baa-a395-eec7327cd99e" containerName="util" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.361984 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="693550bb-21d6-4baa-a395-eec7327cd99e" containerName="util" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.361993 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693550bb-21d6-4baa-a395-eec7327cd99e" containerName="pull" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362000 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="693550bb-21d6-4baa-a395-eec7327cd99e" containerName="pull" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362008 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66b1e1ea-e87e-42e8-b01f-7b721f102073" containerName="util" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362013 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="66b1e1ea-e87e-42e8-b01f-7b721f102073" containerName="util" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362021 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492f19dd-556c-48eb-9573-74d47069bb63" containerName="pull" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362036 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="492f19dd-556c-48eb-9573-74d47069bb63" containerName="pull" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362051 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="921ef911-dbdd-4b2d-8d73-73779c53c166" containerName="util" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362059 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="921ef911-dbdd-4b2d-8d73-73779c53c166" containerName="util" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362067 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693550bb-21d6-4baa-a395-eec7327cd99e" containerName="extract" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362075 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="693550bb-21d6-4baa-a395-eec7327cd99e" containerName="extract" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362084 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66b1e1ea-e87e-42e8-b01f-7b721f102073" containerName="extract" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362091 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="66b1e1ea-e87e-42e8-b01f-7b721f102073" containerName="extract" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362101 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66b1e1ea-e87e-42e8-b01f-7b721f102073" containerName="pull" Sep 11 15:16:51.362093 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362106 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="66b1e1ea-e87e-42e8-b01f-7b721f102073" containerName="pull" Sep 11 15:16:51.362807 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362120 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492f19dd-556c-48eb-9573-74d47069bb63" containerName="util" Sep 11 15:16:51.362807 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362128 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="492f19dd-556c-48eb-9573-74d47069bb63" containerName="util" Sep 11 15:16:51.362807 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362139 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492f19dd-556c-48eb-9573-74d47069bb63" containerName="extract" Sep 11 15:16:51.362807 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362147 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="492f19dd-556c-48eb-9573-74d47069bb63" containerName="extract" Sep 11 15:16:51.362807 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362229 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="66b1e1ea-e87e-42e8-b01f-7b721f102073" containerName="extract" Sep 11 15:16:51.362807 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362238 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="693550bb-21d6-4baa-a395-eec7327cd99e" containerName="extract" Sep 11 15:16:51.362807 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362244 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="492f19dd-556c-48eb-9573-74d47069bb63" containerName="extract" Sep 11 15:16:51.362807 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.362254 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="921ef911-dbdd-4b2d-8d73-73779c53c166" containerName="extract" Sep 11 15:16:51.365191 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.365174 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-sxmbv" Sep 11 15:16:51.367683 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.367655 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 11 15:16:51.367820 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.367659 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 15:16:51.367961 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.367943 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 15:16:51.369518 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.369497 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-j4wv2\"" Sep 11 15:16:51.376046 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.376021 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:16:51.408396 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.408360 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:16:51.445564 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.445519 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br2fb\" (UniqueName: \"kubernetes.io/projected/9676ad60-14ee-411a-a65d-aaff7ed1cf11-kube-api-access-br2fb\") pod \"limitador-limitador-84994f7ffc-sxmbv\" (UID: \"9676ad60-14ee-411a-a65d-aaff7ed1cf11\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sxmbv" Sep 11 15:16:51.445804 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.445599 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9676ad60-14ee-411a-a65d-aaff7ed1cf11-config-file\") pod \"limitador-limitador-84994f7ffc-sxmbv\" (UID: \"9676ad60-14ee-411a-a65d-aaff7ed1cf11\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sxmbv" Sep 11 15:16:51.546088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.546061 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9676ad60-14ee-411a-a65d-aaff7ed1cf11-config-file\") pod \"limitador-limitador-84994f7ffc-sxmbv\" (UID: \"9676ad60-14ee-411a-a65d-aaff7ed1cf11\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sxmbv" Sep 11 15:16:51.546258 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.546136 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br2fb\" (UniqueName: \"kubernetes.io/projected/9676ad60-14ee-411a-a65d-aaff7ed1cf11-kube-api-access-br2fb\") pod \"limitador-limitador-84994f7ffc-sxmbv\" (UID: \"9676ad60-14ee-411a-a65d-aaff7ed1cf11\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sxmbv" Sep 11 15:16:51.546844 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.546824 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9676ad60-14ee-411a-a65d-aaff7ed1cf11-config-file\") pod \"limitador-limitador-84994f7ffc-sxmbv\" (UID: \"9676ad60-14ee-411a-a65d-aaff7ed1cf11\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sxmbv" Sep 11 15:16:51.553958 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.553929 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br2fb\" (UniqueName: \"kubernetes.io/projected/9676ad60-14ee-411a-a65d-aaff7ed1cf11-kube-api-access-br2fb\") pod \"limitador-limitador-84994f7ffc-sxmbv\" (UID: \"9676ad60-14ee-411a-a65d-aaff7ed1cf11\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-sxmbv" Sep 11 15:16:51.676336 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.676300 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-sxmbv" Sep 11 15:16:51.833511 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.833479 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:51.917029 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.916989 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:51.948448 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:51.948422 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:16:51.950135 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:16:51.950101 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9676ad60_14ee_411a_a65d_aaff7ed1cf11.slice/crio-7dc2fd65cdc421114b988ae40f53604f954a9345fd841b29c5790724aebeaba1 WatchSource:0}: Error finding container 7dc2fd65cdc421114b988ae40f53604f954a9345fd841b29c5790724aebeaba1: Status 404 returned error can't find the container with id 7dc2fd65cdc421114b988ae40f53604f954a9345fd841b29c5790724aebeaba1 Sep 11 15:16:52.559634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:52.559599 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-sxmbv" event={"ID":"9676ad60-14ee-411a-a65d-aaff7ed1cf11","Type":"ContainerStarted","Data":"7dc2fd65cdc421114b988ae40f53604f954a9345fd841b29c5790724aebeaba1"} Sep 11 15:16:55.577364 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:55.577329 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-sxmbv" event={"ID":"9676ad60-14ee-411a-a65d-aaff7ed1cf11","Type":"ContainerStarted","Data":"96fdc8b65f2f62ed654d5babb00dc922996941dcd80b68bdbe58534c77aae859"} Sep 11 15:16:55.577859 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:55.577396 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-sxmbv" Sep 11 15:16:55.635383 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:55.635328 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-sxmbv" podStartSLOduration=2.026673017 podStartE2EDuration="4.635310241s" totalImagesPullingTime="2.608637224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:16:55.634164195 +0000 UTC m=+509.866088626" watchObservedRunningTime="2026-09-11 15:16:55.635310241 +0000 UTC m=+509.867234672" Sep 11 15:16:59.259853 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:59.259818 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:16:59.359854 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:16:59.359824 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:01.702622 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:01.702588 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:01.794174 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:01.794128 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:06.458979 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:06.458944 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:06.562225 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:06.562193 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:06.582995 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:06.582974 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-sxmbv" Sep 11 15:17:12.187223 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:12.187171 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:12.296158 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:12.296127 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:12.710619 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:12.710578 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:22.277029 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:22.276951 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:22.565281 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:22.565198 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:22.854289 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:22.854207 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:29.764803 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:29.764770 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:29.861839 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:29.861810 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:30.271045 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:30.271010 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:34.019067 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:34.019029 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:17:34.125488 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:17:34.125454 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:05.780222 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:05.780148 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:05.880172 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:05.880140 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:07.226900 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:07.226859 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:07.327619 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:07.327589 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:07.546842 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:07.546762 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:07.646624 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:07.646591 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:07.866906 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:07.866830 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:07.967293 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:07.967260 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:08.205582 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:08.205553 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:18:19.552612 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:19.552577 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:19.649969 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:19.649937 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:19.990370 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:19.990338 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:46.729239 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:46.729203 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:47.186164 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:47.186124 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:18:57.510865 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:57.510782 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:18:58.004176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:18:58.004146 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:19:09.283503 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.283472 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz"] Sep 11 15:19:09.286800 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.286784 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.289813 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.289794 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 15:19:09.290847 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.290832 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 15:19:09.290847 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.290840 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 11 15:19:09.291371 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.291349 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gfk8t\"" Sep 11 15:19:09.300479 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.300458 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz"] Sep 11 15:19:09.310571 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.310544 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/242f596d-f635-43c0-bc17-3bfdd232eea4-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.310669 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.310591 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/242f596d-f635-43c0-bc17-3bfdd232eea4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.310669 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.310635 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/242f596d-f635-43c0-bc17-3bfdd232eea4-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.310887 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.310669 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d78sz\" (UniqueName: \"kubernetes.io/projected/242f596d-f635-43c0-bc17-3bfdd232eea4-kube-api-access-d78sz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.310887 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.310761 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/242f596d-f635-43c0-bc17-3bfdd232eea4-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.310887 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.310827 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/242f596d-f635-43c0-bc17-3bfdd232eea4-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.330708 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.330665 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:19:09.411485 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.411466 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/242f596d-f635-43c0-bc17-3bfdd232eea4-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.411583 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.411493 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/242f596d-f635-43c0-bc17-3bfdd232eea4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.411583 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.411510 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/242f596d-f635-43c0-bc17-3bfdd232eea4-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.411583 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.411537 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d78sz\" (UniqueName: \"kubernetes.io/projected/242f596d-f635-43c0-bc17-3bfdd232eea4-kube-api-access-d78sz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.411723 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.411684 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/242f596d-f635-43c0-bc17-3bfdd232eea4-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.411818 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.411804 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/242f596d-f635-43c0-bc17-3bfdd232eea4-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.411995 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.411975 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/242f596d-f635-43c0-bc17-3bfdd232eea4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.412065 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.412001 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/242f596d-f635-43c0-bc17-3bfdd232eea4-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.412108 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.412089 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/242f596d-f635-43c0-bc17-3bfdd232eea4-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.413595 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.413580 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/242f596d-f635-43c0-bc17-3bfdd232eea4-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.413928 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.413911 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/242f596d-f635-43c0-bc17-3bfdd232eea4-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.418862 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.418842 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d78sz\" (UniqueName: \"kubernetes.io/projected/242f596d-f635-43c0-bc17-3bfdd232eea4-kube-api-access-d78sz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz\" (UID: \"242f596d-f635-43c0-bc17-3bfdd232eea4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.597139 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.597071 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:09.867652 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.867623 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz"] Sep 11 15:19:09.869445 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:19:09.869420 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242f596d_f635_43c0_bc17_3bfdd232eea4.slice/crio-5dc6c5d044aaa786bac7dad920c85fd1e94dd4373e16834195874b51cdf5e7a0 WatchSource:0}: Error finding container 5dc6c5d044aaa786bac7dad920c85fd1e94dd4373e16834195874b51cdf5e7a0: Status 404 returned error can't find the container with id 5dc6c5d044aaa786bac7dad920c85fd1e94dd4373e16834195874b51cdf5e7a0 Sep 11 15:19:09.870838 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:09.870817 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 15:19:10.247502 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:10.247463 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" event={"ID":"242f596d-f635-43c0-bc17-3bfdd232eea4","Type":"ContainerStarted","Data":"5dc6c5d044aaa786bac7dad920c85fd1e94dd4373e16834195874b51cdf5e7a0"} Sep 11 15:19:10.284009 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:10.283967 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:19:11.254194 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:11.254159 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" event={"ID":"242f596d-f635-43c0-bc17-3bfdd232eea4","Type":"ContainerStarted","Data":"9d8475543b7d86fdf82f6c6cffa1b4ebb2c365b070f127be0b393473be5f13dd"} Sep 11 15:19:11.271928 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:11.271883 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" podStartSLOduration=1.307762139 podStartE2EDuration="2.271867155s" totalImagesPullingTime="964.105016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:19:11.271111801 +0000 UTC m=+645.503036234" watchObservedRunningTime="2026-09-11 15:19:11.271867155 +0000 UTC m=+645.503791597" Sep 11 15:19:12.140133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:12.140096 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:19:13.817987 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:13.817915 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:19:15.203307 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:15.203270 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:19:19.598056 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:19.598016 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:19.598478 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:19.598121 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:19.610030 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:19.610007 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:20.310306 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:20.310278 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz" Sep 11 15:19:21.289040 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.289002 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl"] Sep 11 15:19:21.292494 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.292466 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.294686 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.294664 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 11 15:19:21.300271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.300250 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl"] Sep 11 15:19:21.336417 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.336386 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:19:21.408690 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.408661 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.408872 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.408699 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.408872 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.408717 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.408872 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.408783 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.408872 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.408834 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.409008 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.408885 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2gs\" (UniqueName: \"kubernetes.io/projected/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-kube-api-access-xk2gs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.510036 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.509995 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.510197 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.510046 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.510197 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.510076 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.510197 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.510103 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.510197 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.510145 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.510197 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.510182 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2gs\" (UniqueName: \"kubernetes.io/projected/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-kube-api-access-xk2gs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.510502 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.510464 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.510599 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.510524 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.510647 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.510612 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.512357 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.512333 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.512597 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.512580 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.518554 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.518531 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2gs\" (UniqueName: \"kubernetes.io/projected/5b0a4329-dfd8-4944-9f1b-9a0b53698c1a-kube-api-access-xk2gs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl\" (UID: \"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.603804 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.603708 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:21.877670 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:21.877642 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl"] Sep 11 15:19:21.880352 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:19:21.880326 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0a4329_dfd8_4944_9f1b_9a0b53698c1a.slice/crio-cf0dcf9c3b6a3c2a727b333bf11ad43ad7879a8d161b4c4dd30fbaa1901d10ca WatchSource:0}: Error finding container cf0dcf9c3b6a3c2a727b333bf11ad43ad7879a8d161b4c4dd30fbaa1901d10ca: Status 404 returned error can't find the container with id cf0dcf9c3b6a3c2a727b333bf11ad43ad7879a8d161b4c4dd30fbaa1901d10ca Sep 11 15:19:22.312250 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:22.312204 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" event={"ID":"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a","Type":"ContainerStarted","Data":"110eeb237cde86bb077000337338522c9e54764a38e9cb105007f3b7db911b83"} Sep 11 15:19:22.312617 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:22.312257 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" event={"ID":"5b0a4329-dfd8-4944-9f1b-9a0b53698c1a","Type":"ContainerStarted","Data":"cf0dcf9c3b6a3c2a727b333bf11ad43ad7879a8d161b4c4dd30fbaa1901d10ca"} Sep 11 15:19:22.328146 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:22.328080 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" podStartSLOduration=1.148850992 podStartE2EDuration="1.328062744s" totalImagesPullingTime="179.211752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:19:22.327921851 +0000 UTC m=+656.559846281" watchObservedRunningTime="2026-09-11 15:19:22.328062744 +0000 UTC m=+656.559987175" Sep 11 15:19:22.783852 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:22.783814 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:19:31.604490 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:31.604439 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:31.605104 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:31.604552 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:31.616640 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:31.616610 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:32.370330 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:32.370305 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl" Sep 11 15:19:33.019045 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:33.019013 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:19:34.886123 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:34.886082 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:19:46.660620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:46.660590 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:19:49.319348 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:49.319313 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:19:51.685111 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:51.685077 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:19:57.072856 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:19:57.072822 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:20:43.252625 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:20:43.252589 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:11.875502 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:11.875444 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:11.975872 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:11.975838 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:13.043098 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:13.043054 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:13.142917 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:13.142880 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:29.428433 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:29.428398 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:29.538218 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:29.538179 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:29.783335 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:29.783251 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:29.979366 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:29.979330 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:31.012634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:31.012596 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:31.112978 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:31.112942 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:31.504724 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:31.504689 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:33.404496 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:33.404461 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:21:33.486629 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:33.486592 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:33.579881 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:33.579851 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:34.825507 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:34.825475 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:34.927438 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:34.927407 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:35.051539 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:35.051507 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:35.325428 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:35.325397 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:35.417634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:35.417604 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:37.235725 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:37.235685 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:37.349918 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:37.349879 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:37.492447 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:37.492363 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:37.811631 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:37.811545 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:38.100429 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.100344 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:38.199793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.199759 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:38.469217 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.469179 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk"] Sep 11 15:21:38.472823 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.472801 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.475424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.475400 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-34c71e-openshift-default-dockercfg-sn794\"" Sep 11 15:21:38.487302 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.487268 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk"] Sep 11 15:21:38.508974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.508947 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:38.541059 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.541034 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-data\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.541283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.541077 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv9nh\" (UniqueName: \"kubernetes.io/projected/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-kube-api-access-rv9nh\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.541283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.541106 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istiod-ca-cert\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.541283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.541170 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-credential-socket\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.541283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.541197 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-workload-certs\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.541283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.541238 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-workload-socket\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.541283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.541258 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-podinfo\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.541625 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.541288 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-envoy\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.541625 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.541328 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-token\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.642760 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.642715 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-credential-socket\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.642930 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.642777 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-workload-certs\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.642930 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.642827 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-workload-socket\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.642930 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.642851 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-podinfo\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.642930 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.642867 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-envoy\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.642930 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.642884 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-token\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.642930 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.642929 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-data\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.643300 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.642958 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv9nh\" (UniqueName: \"kubernetes.io/projected/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-kube-api-access-rv9nh\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.643300 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.642998 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istiod-ca-cert\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.643300 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.643184 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-credential-socket\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.643300 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.643224 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-workload-certs\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.643514 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.643369 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-data\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.643572 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.643540 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-workload-socket\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.644067 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.644043 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istiod-ca-cert\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.645457 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.645435 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-podinfo\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.645544 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.645495 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-envoy\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.650678 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.650654 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-token\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.650970 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.650947 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv9nh\" (UniqueName: \"kubernetes.io/projected/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-kube-api-access-rv9nh\") pod \"e2e-autores-34c71e-openshift-default-6b6887f464-sptnk\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:38.750222 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.750133 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:38.784197 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:38.784166 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:39.063125 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:39.063085 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk"] Sep 11 15:21:39.064548 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:21:39.064518 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9d80b0_cdc9_45bd_9d5b_7fc49248ce02.slice/crio-59071d83714fe7b5d1b3cb7175c82c70f00def7e67ad8c1187f2887cf212c6bb WatchSource:0}: Error finding container 59071d83714fe7b5d1b3cb7175c82c70f00def7e67ad8c1187f2887cf212c6bb: Status 404 returned error can't find the container with id 59071d83714fe7b5d1b3cb7175c82c70f00def7e67ad8c1187f2887cf212c6bb Sep 11 15:21:39.177868 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:39.177832 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:39.840643 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:39.840608 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:39.936725 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:39.936681 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:40.027315 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:40.027277 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" event={"ID":"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02","Type":"ContainerStarted","Data":"59071d83714fe7b5d1b3cb7175c82c70f00def7e67ad8c1187f2887cf212c6bb"} Sep 11 15:21:41.107158 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.107113 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t"] Sep 11 15:21:41.114563 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.114536 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.117326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.117277 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6e6b7dd7-serving-cert\"" Sep 11 15:21:41.117326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.117309 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-79rjx\"" Sep 11 15:21:41.117623 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.117543 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:21:41.118323 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.118146 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 15:21:41.118452 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.118412 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 15:21:41.124142 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.124057 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t"] Sep 11 15:21:41.154596 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.154549 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:41.245492 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.245455 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:41.266347 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.266315 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c87694-21d9-4829-b5b9-a4684128fd5e-openshift-service-ca\") pod \"maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.266552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.266382 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-maas-api-tls\") pod \"maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.266552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.266426 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-metrics-certs\") pod \"maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.266552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.266466 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rspl9\" (UniqueName: \"kubernetes.io/projected/94c87694-21d9-4829-b5b9-a4684128fd5e-kube-api-access-rspl9\") pod \"maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.367383 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.367299 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c87694-21d9-4829-b5b9-a4684128fd5e-openshift-service-ca\") pod \"maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.367541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.367395 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-maas-api-tls\") pod \"maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.367541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.367421 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-metrics-certs\") pod \"maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.367541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.367447 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rspl9\" (UniqueName: \"kubernetes.io/projected/94c87694-21d9-4829-b5b9-a4684128fd5e-kube-api-access-rspl9\") pod \"maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.367788 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:21:41.367771 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-serving-cert: secret "maas-api-e2e-ait-6e6b7dd7-serving-cert" not found Sep 11 15:21:41.367857 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:21:41.367846 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-maas-api-tls podName:94c87694-21d9-4829-b5b9-a4684128fd5e nodeName:}" failed. No retries permitted until 2026-09-11 15:21:41.867825715 +0000 UTC m=+796.099750125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-maas-api-tls") pod "maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" (UID: "94c87694-21d9-4829-b5b9-a4684128fd5e") : secret "maas-api-e2e-ait-6e6b7dd7-serving-cert" not found Sep 11 15:21:41.368176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.368140 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c87694-21d9-4829-b5b9-a4684128fd5e-openshift-service-ca\") pod \"maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.370419 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.370399 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-metrics-certs\") pod \"maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.377321 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.377301 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rspl9\" (UniqueName: \"kubernetes.io/projected/94c87694-21d9-4829-b5b9-a4684128fd5e-kube-api-access-rspl9\") pod \"maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.660359 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.660323 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:41.760355 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.760323 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:41.788963 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.788926 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:21:41.789065 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.788999 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:21:41.789065 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.789028 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:21:41.872116 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.872091 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-maas-api-tls\") pod \"maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.874285 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.874261 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-maas-api-tls\") pod \"maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:41.911771 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:41.911659 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:42.032375 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:42.032341 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:42.040393 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:42.040361 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" event={"ID":"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02","Type":"ContainerStarted","Data":"1bb90eaebbf115f2601db0f019d751293d5641178c984dc2925e6aae302e55b8"} Sep 11 15:21:42.059809 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:42.059761 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" podStartSLOduration=1.337516906 podStartE2EDuration="4.059726352s" totalImagesPullingTime="2.722209446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:42.058427509 +0000 UTC m=+796.290351942" watchObservedRunningTime="2026-09-11 15:21:42.059726352 +0000 UTC m=+796.291650783" Sep 11 15:21:42.323656 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:42.323612 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t"] Sep 11 15:21:42.325893 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:21:42.325863 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c87694_21d9_4829_b5b9_a4684128fd5e.slice/crio-171ef2070a815c0b5d1c13834d23d522c3a84a5dddda2f606010a9cf809324bb WatchSource:0}: Error finding container 171ef2070a815c0b5d1c13834d23d522c3a84a5dddda2f606010a9cf809324bb: Status 404 returned error can't find the container with id 171ef2070a815c0b5d1c13834d23d522c3a84a5dddda2f606010a9cf809324bb Sep 11 15:21:42.785135 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:42.785100 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:42.790982 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:42.790961 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:43.049492 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:43.049394 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" event={"ID":"94c87694-21d9-4829-b5b9-a4684128fd5e","Type":"ContainerStarted","Data":"171ef2070a815c0b5d1c13834d23d522c3a84a5dddda2f606010a9cf809324bb"} Sep 11 15:21:43.049781 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:43.049755 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:43.050708 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:43.050675 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:21:43.356985 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:43.356908 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:44.045252 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:44.045219 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:44.149635 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:44.149598 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:44.590772 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:44.590735 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:45.063443 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:45.063411 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" event={"ID":"94c87694-21d9-4829-b5b9-a4684128fd5e","Type":"ContainerStarted","Data":"08b508479591007ff3af1135e28d49d7acd5638e1e8780c8cb55ed7e28d49fd0"} Sep 11 15:21:45.063635 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:45.063490 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:45.752355 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:45.752320 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:46.580354 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:46.580326 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:46.679198 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:46.679171 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:46.923845 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:46.923820 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:47.077968 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:47.077932 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:47.172498 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:47.172444 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:47.518754 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:47.518720 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:48.141558 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:48.141523 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:48.240802 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:48.240768 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:48.488100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:48.488019 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:48.896634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:48.896610 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:48.986732 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:48.986694 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:49.087175 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:49.087138 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:49.191547 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:49.191471 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:49.285861 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:49.285828 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:49.386755 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:49.386711 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:49.908361 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:49.908324 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:50.289186 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:50.289087 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" podStartSLOduration=7.053362075 podStartE2EDuration="9.289071011s" totalImagesPullingTime="2.235708936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:45.112227238 +0000 UTC m=+799.344151694" watchObservedRunningTime="2026-09-11 15:21:50.289071011 +0000 UTC m=+804.520995489" Sep 11 15:21:50.290766 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:50.290734 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:21:51.074082 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:51.074053 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:21:51.912224 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:51.912189 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:52.011532 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:52.011501 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:52.666902 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:52.666867 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:52.762865 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:52.762831 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:53.345363 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:53.345330 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:53.660828 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:53.660788 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:54.427934 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:54.427903 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:54.741668 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:54.741580 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:55.665575 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:55.665527 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:56.901958 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:56.901926 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x"] Sep 11 15:21:56.905883 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:56.905860 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:56.909165 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:56.909142 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9f08bd-serving-cert\"" Sep 11 15:21:56.919363 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:56.919298 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x"] Sep 11 15:21:56.945557 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:56.945525 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:57.012171 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.012139 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:57.015307 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.015275 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-maas-api-tls\") pod \"maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:57.015460 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.015361 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7m6j\" (UniqueName: \"kubernetes.io/projected/1f410641-3c6f-40e7-b8f7-c4d05b623974-kube-api-access-c7m6j\") pod \"maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:57.015591 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.015480 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f410641-3c6f-40e7-b8f7-c4d05b623974-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:57.015591 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.015508 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-metrics-certs\") pod \"maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:57.098156 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.098121 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:57.116642 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.116607 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f410641-3c6f-40e7-b8f7-c4d05b623974-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:57.116642 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.116650 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-metrics-certs\") pod \"maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:57.116935 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.116694 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-maas-api-tls\") pod \"maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:57.116935 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.116762 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7m6j\" (UniqueName: \"kubernetes.io/projected/1f410641-3c6f-40e7-b8f7-c4d05b623974-kube-api-access-c7m6j\") pod \"maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:57.116935 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:21:57.116846 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-serving-cert: secret "maas-api-e2e-inf-b-9f08bd-serving-cert" not found Sep 11 15:21:57.117066 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:21:57.116944 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-maas-api-tls podName:1f410641-3c6f-40e7-b8f7-c4d05b623974 nodeName:}" failed. No retries permitted until 2026-09-11 15:21:57.61691997 +0000 UTC m=+811.848844395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-maas-api-tls") pod "maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" (UID: "1f410641-3c6f-40e7-b8f7-c4d05b623974") : secret "maas-api-e2e-inf-b-9f08bd-serving-cert" not found Sep 11 15:21:57.117418 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.117400 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f410641-3c6f-40e7-b8f7-c4d05b623974-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:57.119361 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.119308 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-metrics-certs\") pod \"maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:57.129577 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.129546 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7m6j\" (UniqueName: \"kubernetes.io/projected/1f410641-3c6f-40e7-b8f7-c4d05b623974-kube-api-access-c7m6j\") pod \"maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:57.296721 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.296628 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:57.394803 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.394766 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:57.553079 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.552848 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:57.620848 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.620815 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-maas-api-tls\") pod \"maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:57.623249 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.623232 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-maas-api-tls\") pod \"maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:57.647599 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.647572 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:57.819375 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:57.819285 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:58.061766 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:58.059353 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:58.121059 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:58.121033 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x"] Sep 11 15:21:58.123132 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:21:58.123104 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f410641_3c6f_40e7_b8f7_c4d05b623974.slice/crio-c64ed527d495d030bcea7c41b368816aa7a627c1ff1149035625c56343c1018d WatchSource:0}: Error finding container c64ed527d495d030bcea7c41b368816aa7a627c1ff1149035625c56343c1018d: Status 404 returned error can't find the container with id c64ed527d495d030bcea7c41b368816aa7a627c1ff1149035625c56343c1018d Sep 11 15:21:58.135529 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:58.135505 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" event={"ID":"1f410641-3c6f-40e7-b8f7-c4d05b623974","Type":"ContainerStarted","Data":"c64ed527d495d030bcea7c41b368816aa7a627c1ff1149035625c56343c1018d"} Sep 11 15:21:58.768015 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:58.767985 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:58.916629 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:58.916595 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:21:59.144526 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:59.144477 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" event={"ID":"1f410641-3c6f-40e7-b8f7-c4d05b623974","Type":"ContainerStarted","Data":"61efe59b224e2f93f63815a5be11e5a0bd2d73648a8339deb4e95c6c04b1a6b5"} Sep 11 15:21:59.144921 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:59.144581 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:21:59.165838 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:59.165787 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" podStartSLOduration=2.775879195 podStartE2EDuration="3.165769685s" totalImagesPullingTime="389.89049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:21:59.162720982 +0000 UTC m=+813.394645432" watchObservedRunningTime="2026-09-11 15:21:59.165769685 +0000 UTC m=+813.397694116" Sep 11 15:21:59.696769 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:21:59.696725 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:00.466097 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:00.466066 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:01.411911 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:01.411872 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:02.161924 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:02.161893 2558 generic.go:356] "Generic (PLEG): container finished" podID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerID="1bb90eaebbf115f2601db0f019d751293d5641178c984dc2925e6aae302e55b8" exitCode=0 Sep 11 15:22:02.162316 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:02.161966 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" event={"ID":"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02","Type":"ContainerDied","Data":"1bb90eaebbf115f2601db0f019d751293d5641178c984dc2925e6aae302e55b8"} Sep 11 15:22:02.162369 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:02.162330 2558 scope.go:117] "RemoveContainer" containerID="1bb90eaebbf115f2601db0f019d751293d5641178c984dc2925e6aae302e55b8" Sep 11 15:22:02.163445 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:02.163418 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:22:02.163505 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:02.163486 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:22:02.163545 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:02.163518 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:22:02.232901 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:02.232876 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:03.036787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:03.036689 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:03.174034 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:03.173984 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" event={"ID":"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02","Type":"ContainerStarted","Data":"62aebcb075ebc4571a03730fd6a2ea3db38206e02fc30e2da3deff4cb0140bd8"} Sep 11 15:22:03.784794 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:03.784724 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:22:03.785061 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:03.785032 2558 patch_prober.go:28] interesting pod/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 11 15:22:03.785155 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:03.785093 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" podUID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 11 15:22:04.099889 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:04.099791 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:04.216958 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:04.216925 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:04.323430 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:04.323398 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:04.430381 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:04.430345 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:04.536668 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:04.536612 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:04.785243 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:04.785150 2558 patch_prober.go:28] interesting pod/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 11 15:22:04.785243 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:04.785209 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" podUID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 11 15:22:05.167032 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:05.166998 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:22:05.785120 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:05.785074 2558 patch_prober.go:28] interesting pod/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 11 15:22:05.785498 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:05.785136 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" podUID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 11 15:22:05.818602 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:05.818567 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:06.785427 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:06.785384 2558 patch_prober.go:28] interesting pod/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 11 15:22:06.785852 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:06.785450 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" podUID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 11 15:22:07.784599 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:07.784567 2558 patch_prober.go:28] interesting pod/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 11 15:22:07.784862 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:07.784834 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" podUID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 11 15:22:08.788128 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:08.788097 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:22:08.788590 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:08.788473 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:22:08.789249 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:08.789230 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:22:10.343787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:10.343731 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:10.445641 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:10.445612 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:10.621928 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:10.621840 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:11.187300 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.187266 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t"] Sep 11 15:22:11.187555 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.187515 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" podUID="94c87694-21d9-4829-b5b9-a4684128fd5e" containerName="maas-api" containerID="cri-o://08b508479591007ff3af1135e28d49d7acd5638e1e8780c8cb55ed7e28d49fd0" gracePeriod=30 Sep 11 15:22:11.223845 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.223816 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:11.328290 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.328250 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:11.426396 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.426372 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:11.433506 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.433482 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:22:11.526211 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.526024 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:11.541080 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.541043 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-metrics-certs\") pod \"94c87694-21d9-4829-b5b9-a4684128fd5e\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " Sep 11 15:22:11.541268 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.541156 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rspl9\" (UniqueName: \"kubernetes.io/projected/94c87694-21d9-4829-b5b9-a4684128fd5e-kube-api-access-rspl9\") pod \"94c87694-21d9-4829-b5b9-a4684128fd5e\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " Sep 11 15:22:11.541268 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.541191 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-maas-api-tls\") pod \"94c87694-21d9-4829-b5b9-a4684128fd5e\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " Sep 11 15:22:11.541268 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.541219 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c87694-21d9-4829-b5b9-a4684128fd5e-openshift-service-ca\") pod \"94c87694-21d9-4829-b5b9-a4684128fd5e\" (UID: \"94c87694-21d9-4829-b5b9-a4684128fd5e\") " Sep 11 15:22:11.541857 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.541823 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c87694-21d9-4829-b5b9-a4684128fd5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94c87694-21d9-4829-b5b9-a4684128fd5e" (UID: "94c87694-21d9-4829-b5b9-a4684128fd5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:22:11.544223 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.544196 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "94c87694-21d9-4829-b5b9-a4684128fd5e" (UID: "94c87694-21d9-4829-b5b9-a4684128fd5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:22:11.545496 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.545467 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94c87694-21d9-4829-b5b9-a4684128fd5e" (UID: "94c87694-21d9-4829-b5b9-a4684128fd5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:22:11.545670 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.545647 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c87694-21d9-4829-b5b9-a4684128fd5e-kube-api-access-rspl9" (OuterVolumeSpecName: "kube-api-access-rspl9") pod "94c87694-21d9-4829-b5b9-a4684128fd5e" (UID: "94c87694-21d9-4829-b5b9-a4684128fd5e"). InnerVolumeSpecName "kube-api-access-rspl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:22:11.635263 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.635236 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:11.642409 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.642390 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rspl9\" (UniqueName: \"kubernetes.io/projected/94c87694-21d9-4829-b5b9-a4684128fd5e-kube-api-access-rspl9\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:11.642493 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.642411 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:11.642493 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.642420 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94c87694-21d9-4829-b5b9-a4684128fd5e-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:11.642493 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.642429 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94c87694-21d9-4829-b5b9-a4684128fd5e-metrics-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:11.910176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:11.910126 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:12.070725 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.070692 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:12.164814 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.164680 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:12.222291 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.222264 2558 generic.go:356] "Generic (PLEG): container finished" podID="94c87694-21d9-4829-b5b9-a4684128fd5e" containerID="08b508479591007ff3af1135e28d49d7acd5638e1e8780c8cb55ed7e28d49fd0" exitCode=0 Sep 11 15:22:12.222421 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.222307 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" Sep 11 15:22:12.222421 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.222348 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" event={"ID":"94c87694-21d9-4829-b5b9-a4684128fd5e","Type":"ContainerDied","Data":"08b508479591007ff3af1135e28d49d7acd5638e1e8780c8cb55ed7e28d49fd0"} Sep 11 15:22:12.222421 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.222393 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t" event={"ID":"94c87694-21d9-4829-b5b9-a4684128fd5e","Type":"ContainerDied","Data":"171ef2070a815c0b5d1c13834d23d522c3a84a5dddda2f606010a9cf809324bb"} Sep 11 15:22:12.222421 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.222415 2558 scope.go:117] "RemoveContainer" containerID="08b508479591007ff3af1135e28d49d7acd5638e1e8780c8cb55ed7e28d49fd0" Sep 11 15:22:12.232108 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.232091 2558 scope.go:117] "RemoveContainer" containerID="08b508479591007ff3af1135e28d49d7acd5638e1e8780c8cb55ed7e28d49fd0" Sep 11 15:22:12.232392 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:22:12.232373 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b508479591007ff3af1135e28d49d7acd5638e1e8780c8cb55ed7e28d49fd0\": container with ID starting with 08b508479591007ff3af1135e28d49d7acd5638e1e8780c8cb55ed7e28d49fd0 not found: ID does not exist" containerID="08b508479591007ff3af1135e28d49d7acd5638e1e8780c8cb55ed7e28d49fd0" Sep 11 15:22:12.232440 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.232399 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b508479591007ff3af1135e28d49d7acd5638e1e8780c8cb55ed7e28d49fd0"} err="failed to get container status \"08b508479591007ff3af1135e28d49d7acd5638e1e8780c8cb55ed7e28d49fd0\": rpc error: code = NotFound desc = could not find container \"08b508479591007ff3af1135e28d49d7acd5638e1e8780c8cb55ed7e28d49fd0\": container with ID starting with 08b508479591007ff3af1135e28d49d7acd5638e1e8780c8cb55ed7e28d49fd0 not found: ID does not exist" Sep 11 15:22:12.244850 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.244828 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t"] Sep 11 15:22:12.248984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.248967 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e6b7dd7-6bfb44b456-gsk5t"] Sep 11 15:22:12.264024 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.264003 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94c87694-21d9-4829-b5b9-a4684128fd5e" path="/var/lib/kubelet/pods/94c87694-21d9-4829-b5b9-a4684128fd5e/volumes" Sep 11 15:22:12.280074 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.280050 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:12.610826 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.610727 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k"] Sep 11 15:22:12.611176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.611096 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c87694-21d9-4829-b5b9-a4684128fd5e" containerName="maas-api" Sep 11 15:22:12.611176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.611108 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c87694-21d9-4829-b5b9-a4684128fd5e" containerName="maas-api" Sep 11 15:22:12.611250 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.611181 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="94c87694-21d9-4829-b5b9-a4684128fd5e" containerName="maas-api" Sep 11 15:22:12.615450 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.615430 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.618431 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.618412 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-73b24789-gw-openshift-default-dockercfg-s6t77\"" Sep 11 15:22:12.627639 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.627619 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k"] Sep 11 15:22:12.635930 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.635906 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:12.651796 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.651774 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfn6v\" (UniqueName: \"kubernetes.io/projected/d5833fd5-d59b-466e-8590-2210e73d3a1e-kube-api-access-gfn6v\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.651952 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.651881 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5833fd5-d59b-466e-8590-2210e73d3a1e-istiod-ca-cert\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.651952 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.651911 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-podinfo\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.652063 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.651974 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-data\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.652063 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.652046 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-workload-certs\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.652179 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.652085 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-token\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.652179 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.652122 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-workload-socket\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.652179 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.652160 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-envoy\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.652319 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.652201 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-credential-socket\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.753631 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.753594 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-envoy\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.753846 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.753642 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-credential-socket\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.753846 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.753672 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfn6v\" (UniqueName: \"kubernetes.io/projected/d5833fd5-d59b-466e-8590-2210e73d3a1e-kube-api-access-gfn6v\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.753846 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.753757 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5833fd5-d59b-466e-8590-2210e73d3a1e-istiod-ca-cert\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.753846 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.753788 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-podinfo\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.753846 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.753816 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-data\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.753846 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.753845 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-workload-certs\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.754193 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.753875 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-token\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.754193 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.753908 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-workload-socket\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.754193 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.754079 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-credential-socket\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.754367 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.754258 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-workload-socket\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.754367 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.754284 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-workload-certs\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.754367 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.754346 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-data\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.754527 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.754464 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5833fd5-d59b-466e-8590-2210e73d3a1e-istiod-ca-cert\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.756112 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.756087 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-envoy\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.756445 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.756426 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-podinfo\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.761043 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.761025 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-token\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.761321 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.761303 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfn6v\" (UniqueName: \"kubernetes.io/projected/d5833fd5-d59b-466e-8590-2210e73d3a1e-kube-api-access-gfn6v\") pod \"e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:12.809637 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.809610 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:12.926574 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:12.926544 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:13.190527 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:13.190450 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:13.210415 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:13.210384 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k"] Sep 11 15:22:13.210536 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:22:13.210494 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5833fd5_d59b_466e_8590_2210e73d3a1e.slice/crio-9565846b73190d4956f24bcdfb19bebdc554dbfbe7b0c4b9fa1b426833a7a800 WatchSource:0}: Error finding container 9565846b73190d4956f24bcdfb19bebdc554dbfbe7b0c4b9fa1b426833a7a800: Status 404 returned error can't find the container with id 9565846b73190d4956f24bcdfb19bebdc554dbfbe7b0c4b9fa1b426833a7a800 Sep 11 15:22:13.212847 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:13.212817 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:22:13.212909 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:13.212878 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:22:13.212946 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:13.212910 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:22:13.230332 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:13.230290 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" event={"ID":"d5833fd5-d59b-466e-8590-2210e73d3a1e","Type":"ContainerStarted","Data":"9565846b73190d4956f24bcdfb19bebdc554dbfbe7b0c4b9fa1b426833a7a800"} Sep 11 15:22:14.095311 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:14.095274 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:22:14.238378 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:14.238344 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" event={"ID":"d5833fd5-d59b-466e-8590-2210e73d3a1e","Type":"ContainerStarted","Data":"df525ad7fa5ba3956bdb4f3a089406acb55693e39e5d10784101a3d5f9c5bc29"} Sep 11 15:22:14.306115 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:14.306067 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" podStartSLOduration=2.306051597 podStartE2EDuration="2.306051597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:22:14.303755556 +0000 UTC m=+828.535679980" watchObservedRunningTime="2026-09-11 15:22:14.306051597 +0000 UTC m=+828.537976028" Sep 11 15:22:14.370819 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:14.370715 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:14.927506 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:14.927471 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:14.932724 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:14.932698 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:15.243933 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:15.243839 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:15.244696 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:15.244673 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:15.314379 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:15.314346 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:15.885058 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:15.884999 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:15.982769 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:15.982724 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:16.316871 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:16.316838 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:17.143172 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:17.143143 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:17.482075 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:17.481983 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:17.905027 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:17.904995 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx"] Sep 11 15:22:17.909338 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:17.908699 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:17.911549 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:17.911526 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-73b24789-serving-cert\"" Sep 11 15:22:17.923111 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:17.923089 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx"] Sep 11 15:22:17.937767 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:17.937723 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:18.002646 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.002613 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:18.002782 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.002644 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-maas-api-tls\") pod \"maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:18.002782 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.002766 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-metrics-certs\") pod \"maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:18.002907 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.002796 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-openshift-service-ca\") pod \"maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:18.002907 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.002821 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gltcj\" (UniqueName: \"kubernetes.io/projected/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-kube-api-access-gltcj\") pod \"maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:18.103958 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.103917 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-metrics-certs\") pod \"maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:18.103958 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.103964 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-openshift-service-ca\") pod \"maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:18.104227 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.104095 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gltcj\" (UniqueName: \"kubernetes.io/projected/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-kube-api-access-gltcj\") pod \"maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:18.104227 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.104188 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-maas-api-tls\") pod \"maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:18.104339 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:22:18.104293 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-serving-cert: secret "maas-api-e2e-migrate-73b24789-serving-cert" not found Sep 11 15:22:18.104396 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:22:18.104357 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-maas-api-tls podName:03f4297f-e5b6-4a5f-a897-02cc2e851bcf nodeName:}" failed. No retries permitted until 2026-09-11 15:22:18.604340542 +0000 UTC m=+832.836264950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-maas-api-tls") pod "maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" (UID: "03f4297f-e5b6-4a5f-a897-02cc2e851bcf") : secret "maas-api-e2e-migrate-73b24789-serving-cert" not found Sep 11 15:22:18.104722 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.104703 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-openshift-service-ca\") pod \"maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:18.106577 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.106549 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-metrics-certs\") pod \"maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:18.113086 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.113062 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gltcj\" (UniqueName: \"kubernetes.io/projected/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-kube-api-access-gltcj\") pod \"maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:18.168845 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.168733 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:18.520620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.520536 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:18.610286 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.610243 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-maas-api-tls\") pod \"maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:18.612655 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.612638 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-maas-api-tls\") pod \"maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:18.620872 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.620848 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:18.822885 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:18.822803 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:19.099249 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:19.099222 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx"] Sep 11 15:22:19.099995 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:22:19.099955 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f4297f_e5b6_4a5f_a897_02cc2e851bcf.slice/crio-08f7c495e8b69ba10f013b74db60f8ba421fabcfc20da64208d1719109657198 WatchSource:0}: Error finding container 08f7c495e8b69ba10f013b74db60f8ba421fabcfc20da64208d1719109657198: Status 404 returned error can't find the container with id 08f7c495e8b69ba10f013b74db60f8ba421fabcfc20da64208d1719109657198 Sep 11 15:22:19.161708 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:19.161667 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:19.266039 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:19.266001 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" event={"ID":"03f4297f-e5b6-4a5f-a897-02cc2e851bcf","Type":"ContainerStarted","Data":"08f7c495e8b69ba10f013b74db60f8ba421fabcfc20da64208d1719109657198"} Sep 11 15:22:20.015504 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:20.015469 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:20.272585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:20.272496 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" event={"ID":"03f4297f-e5b6-4a5f-a897-02cc2e851bcf","Type":"ContainerStarted","Data":"d4fd42fa88bd9db7c59bc094e6510ccdbd55eaf97912eb37fdbabf0bbce031eb"} Sep 11 15:22:20.272777 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:20.272648 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:20.293258 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:20.293211 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" podStartSLOduration=2.919822708 podStartE2EDuration="3.293196778s" totalImagesPullingTime="373.37407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:22:20.291788587 +0000 UTC m=+834.523713020" watchObservedRunningTime="2026-09-11 15:22:20.293196778 +0000 UTC m=+834.525121210" Sep 11 15:22:20.811512 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:20.811481 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:23.593554 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:23.593524 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:26.283820 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:26.283792 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:31.284877 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:31.284840 2558 patch_prober.go:28] interesting pod/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 15:22:31.285238 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:31.284897 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 15:22:31.390358 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:31.390318 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:22:32.338730 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:32.338637 2558 generic.go:356] "Generic (PLEG): container finished" podID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerID="df525ad7fa5ba3956bdb4f3a089406acb55693e39e5d10784101a3d5f9c5bc29" exitCode=0 Sep 11 15:22:32.338730 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:32.338711 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" event={"ID":"d5833fd5-d59b-466e-8590-2210e73d3a1e","Type":"ContainerDied","Data":"df525ad7fa5ba3956bdb4f3a089406acb55693e39e5d10784101a3d5f9c5bc29"} Sep 11 15:22:32.339188 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:32.339168 2558 scope.go:117] "RemoveContainer" containerID="df525ad7fa5ba3956bdb4f3a089406acb55693e39e5d10784101a3d5f9c5bc29" Sep 11 15:22:32.340298 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:32.340265 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:22:32.340399 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:32.340347 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:22:32.340464 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:32.340397 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:22:33.346677 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:33.346645 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" event={"ID":"d5833fd5-d59b-466e-8590-2210e73d3a1e","Type":"ContainerStarted","Data":"05e257c89dc018baa52b70a199014ee695012bf2b689450593ca4aea1fa5a583"} Sep 11 15:22:33.927086 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:33.927048 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:33.927266 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:33.927236 2558 patch_prober.go:28] interesting pod/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 11 15:22:33.927312 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:33.927284 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 11 15:22:34.926972 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:34.926933 2558 patch_prober.go:28] interesting pod/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 11 15:22:34.927338 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:34.926991 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 11 15:22:35.927329 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:35.927285 2558 patch_prober.go:28] interesting pod/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 11 15:22:35.927771 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:35.927348 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 11 15:22:36.926965 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:36.926926 2558 patch_prober.go:28] interesting pod/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 11 15:22:36.927147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:36.927009 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 11 15:22:37.927935 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:37.927895 2558 patch_prober.go:28] interesting pod/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Sep 11 15:22:37.928324 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:37.927954 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Sep 11 15:22:38.931490 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:38.931457 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:38.931893 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:38.931773 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:38.932577 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:38.932557 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:40.226920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:40.226884 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:41.105962 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:41.105925 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:41.811251 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:41.811222 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:42.544784 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:42.544724 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:43.226541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:43.226506 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:44.150275 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:44.150241 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:45.155482 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:45.155406 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:47.136346 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:47.136314 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:47.813252 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:47.813220 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx"] Sep 11 15:22:47.813470 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:47.813444 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" podUID="03f4297f-e5b6-4a5f-a897-02cc2e851bcf" containerName="maas-api" containerID="cri-o://d4fd42fa88bd9db7c59bc094e6510ccdbd55eaf97912eb37fdbabf0bbce031eb" gracePeriod=30 Sep 11 15:22:47.838957 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:47.838923 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:47.912286 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:47.912253 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:48.015001 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.014970 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:48.058847 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.058798 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:48.094989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.094915 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:48.104028 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.104004 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-openshift-service-ca\") pod \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " Sep 11 15:22:48.104107 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.104097 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-metrics-certs\") pod \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " Sep 11 15:22:48.104205 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.104190 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gltcj\" (UniqueName: \"kubernetes.io/projected/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-kube-api-access-gltcj\") pod \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " Sep 11 15:22:48.104248 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.104222 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-maas-api-tls\") pod \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\" (UID: \"03f4297f-e5b6-4a5f-a897-02cc2e851bcf\") " Sep 11 15:22:48.104391 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.104364 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03f4297f-e5b6-4a5f-a897-02cc2e851bcf" (UID: "03f4297f-e5b6-4a5f-a897-02cc2e851bcf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:22:48.104524 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.104509 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:48.106293 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.106269 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03f4297f-e5b6-4a5f-a897-02cc2e851bcf" (UID: "03f4297f-e5b6-4a5f-a897-02cc2e851bcf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:22:48.106396 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.106306 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "03f4297f-e5b6-4a5f-a897-02cc2e851bcf" (UID: "03f4297f-e5b6-4a5f-a897-02cc2e851bcf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:22:48.106396 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.106341 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-kube-api-access-gltcj" (OuterVolumeSpecName: "kube-api-access-gltcj") pod "03f4297f-e5b6-4a5f-a897-02cc2e851bcf" (UID: "03f4297f-e5b6-4a5f-a897-02cc2e851bcf"). InnerVolumeSpecName "kube-api-access-gltcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:22:48.205244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.205208 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-metrics-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:48.205244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.205244 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gltcj\" (UniqueName: \"kubernetes.io/projected/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-kube-api-access-gltcj\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:48.205855 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.205260 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f4297f-e5b6-4a5f-a897-02cc2e851bcf-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:48.230189 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.230159 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k"] Sep 11 15:22:48.230370 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.230350 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerName="istio-proxy" containerID="cri-o://05e257c89dc018baa52b70a199014ee695012bf2b689450593ca4aea1fa5a583" gracePeriod=30 Sep 11 15:22:48.260150 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.260119 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:48.440997 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.440960 2558 generic.go:356] "Generic (PLEG): container finished" podID="03f4297f-e5b6-4a5f-a897-02cc2e851bcf" containerID="d4fd42fa88bd9db7c59bc094e6510ccdbd55eaf97912eb37fdbabf0bbce031eb" exitCode=0 Sep 11 15:22:48.441202 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.441037 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" event={"ID":"03f4297f-e5b6-4a5f-a897-02cc2e851bcf","Type":"ContainerDied","Data":"d4fd42fa88bd9db7c59bc094e6510ccdbd55eaf97912eb37fdbabf0bbce031eb"} Sep 11 15:22:48.441265 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.441237 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" event={"ID":"03f4297f-e5b6-4a5f-a897-02cc2e851bcf","Type":"ContainerDied","Data":"08f7c495e8b69ba10f013b74db60f8ba421fabcfc20da64208d1719109657198"} Sep 11 15:22:48.441309 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.441085 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx" Sep 11 15:22:48.441366 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.441264 2558 scope.go:117] "RemoveContainer" containerID="d4fd42fa88bd9db7c59bc094e6510ccdbd55eaf97912eb37fdbabf0bbce031eb" Sep 11 15:22:48.450650 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.450632 2558 scope.go:117] "RemoveContainer" containerID="d4fd42fa88bd9db7c59bc094e6510ccdbd55eaf97912eb37fdbabf0bbce031eb" Sep 11 15:22:48.450961 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:22:48.450940 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fd42fa88bd9db7c59bc094e6510ccdbd55eaf97912eb37fdbabf0bbce031eb\": container with ID starting with d4fd42fa88bd9db7c59bc094e6510ccdbd55eaf97912eb37fdbabf0bbce031eb not found: ID does not exist" containerID="d4fd42fa88bd9db7c59bc094e6510ccdbd55eaf97912eb37fdbabf0bbce031eb" Sep 11 15:22:48.451031 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.450969 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fd42fa88bd9db7c59bc094e6510ccdbd55eaf97912eb37fdbabf0bbce031eb"} err="failed to get container status \"d4fd42fa88bd9db7c59bc094e6510ccdbd55eaf97912eb37fdbabf0bbce031eb\": rpc error: code = NotFound desc = could not find container \"d4fd42fa88bd9db7c59bc094e6510ccdbd55eaf97912eb37fdbabf0bbce031eb\": container with ID starting with d4fd42fa88bd9db7c59bc094e6510ccdbd55eaf97912eb37fdbabf0bbce031eb not found: ID does not exist" Sep 11 15:22:48.462326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.462299 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx"] Sep 11 15:22:48.467241 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.467211 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-73b24789-f54b4d66b-r68jx"] Sep 11 15:22:48.519639 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.519602 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:48.738454 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:48.735364 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:49.048993 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.048910 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld"] Sep 11 15:22:49.049424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.049406 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f4297f-e5b6-4a5f-a897-02cc2e851bcf" containerName="maas-api" Sep 11 15:22:49.049508 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.049427 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f4297f-e5b6-4a5f-a897-02cc2e851bcf" containerName="maas-api" Sep 11 15:22:49.049508 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.049501 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f4297f-e5b6-4a5f-a897-02cc2e851bcf" containerName="maas-api" Sep 11 15:22:49.054175 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.054152 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.056879 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.056857 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8c097d73-openshift-default-dockercfg-9x7l6\"" Sep 11 15:22:49.064967 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.064942 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld"] Sep 11 15:22:49.087808 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.087778 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:49.114578 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.114551 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-data\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.114722 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.114589 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2fng\" (UniqueName: \"kubernetes.io/projected/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-kube-api-access-w2fng\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.114722 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.114616 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-token\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.114722 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.114701 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-envoy\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.114897 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.114787 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-podinfo\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.114897 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.114816 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-workload-certs\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.114897 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.114839 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istiod-ca-cert\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.114897 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.114856 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-credential-socket\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.115097 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.114934 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-workload-socket\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.216267 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.216234 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-podinfo\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.216267 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.216265 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-workload-certs\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.216816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.216288 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istiod-ca-cert\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.216816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.216305 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-credential-socket\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.216816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.216340 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-workload-socket\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.216816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.216379 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-data\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.216816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.216399 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2fng\" (UniqueName: \"kubernetes.io/projected/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-kube-api-access-w2fng\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.216816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.216414 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-token\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.216816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.216454 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-envoy\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.216816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.216722 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-credential-socket\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.216816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.216727 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-workload-certs\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.217177 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.216832 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-data\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.217177 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.216895 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-workload-socket\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.217512 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.217489 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istiod-ca-cert\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.218669 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.218640 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-envoy\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.219027 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.219007 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-podinfo\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.223756 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.223712 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2fng\" (UniqueName: \"kubernetes.io/projected/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-kube-api-access-w2fng\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.223852 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.223716 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-token\") pod \"e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.283543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.283513 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:49.366039 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.365956 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:49.392368 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.392333 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:49.651623 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.651592 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld"] Sep 11 15:22:49.653311 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:22:49.653276 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cb18c7_a74b_42ac_b515_1e8c4f5c603a.slice/crio-d641459e2e127c5cba22eccf3b106280d9d86dbd5ce065f16b2e136395088918 WatchSource:0}: Error finding container d641459e2e127c5cba22eccf3b106280d9d86dbd5ce065f16b2e136395088918: Status 404 returned error can't find the container with id d641459e2e127c5cba22eccf3b106280d9d86dbd5ce065f16b2e136395088918 Sep 11 15:22:49.656183 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.656140 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:22:49.656286 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.656250 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:22:49.656335 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.656296 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:22:49.690237 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:49.690211 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:50.044592 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:50.044506 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:50.264468 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:50.264434 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f4297f-e5b6-4a5f-a897-02cc2e851bcf" path="/var/lib/kubelet/pods/03f4297f-e5b6-4a5f-a897-02cc2e851bcf/volumes" Sep 11 15:22:50.455631 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:50.455593 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" event={"ID":"77cb18c7-a74b-42ac-b515-1e8c4f5c603a","Type":"ContainerStarted","Data":"1a3b7077b97a8b1668594940a7d48574c8135e9597b531fa8835823f8e3a0446"} Sep 11 15:22:50.455864 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:50.455657 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" event={"ID":"77cb18c7-a74b-42ac-b515-1e8c4f5c603a","Type":"ContainerStarted","Data":"d641459e2e127c5cba22eccf3b106280d9d86dbd5ce065f16b2e136395088918"} Sep 11 15:22:50.500530 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:50.500477 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" podStartSLOduration=1.500460015 podStartE2EDuration="1.500460015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:22:50.499107211 +0000 UTC m=+864.731031643" watchObservedRunningTime="2026-09-11 15:22:50.500460015 +0000 UTC m=+864.732384446" Sep 11 15:22:50.652644 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:50.652612 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:22:51.015733 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:51.015699 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:51.366449 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:51.366367 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:51.372577 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:51.372551 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:51.461632 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:51.461603 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:51.462578 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:51.462559 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:22:51.834346 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:51.834313 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:52.014487 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:52.014449 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:52.110711 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:52.110636 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:52.904247 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:52.904209 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:53.473760 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.473706 2558 generic.go:356] "Generic (PLEG): container finished" podID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerID="05e257c89dc018baa52b70a199014ee695012bf2b689450593ca4aea1fa5a583" exitCode=0 Sep 11 15:22:53.473887 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.473771 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" event={"ID":"d5833fd5-d59b-466e-8590-2210e73d3a1e","Type":"ContainerDied","Data":"05e257c89dc018baa52b70a199014ee695012bf2b689450593ca4aea1fa5a583"} Sep 11 15:22:53.473887 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.473829 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" event={"ID":"d5833fd5-d59b-466e-8590-2210e73d3a1e","Type":"ContainerDied","Data":"9565846b73190d4956f24bcdfb19bebdc554dbfbe7b0c4b9fa1b426833a7a800"} Sep 11 15:22:53.473887 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.473849 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9565846b73190d4956f24bcdfb19bebdc554dbfbe7b0c4b9fa1b426833a7a800" Sep 11 15:22:53.473887 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.473863 2558 scope.go:117] "RemoveContainer" containerID="df525ad7fa5ba3956bdb4f3a089406acb55693e39e5d10784101a3d5f9c5bc29" Sep 11 15:22:53.481630 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.481611 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:53.553379 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.553306 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-credential-socket\") pod \"d5833fd5-d59b-466e-8590-2210e73d3a1e\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " Sep 11 15:22:53.553379 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.553359 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-envoy\") pod \"d5833fd5-d59b-466e-8590-2210e73d3a1e\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " Sep 11 15:22:53.553581 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.553392 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5833fd5-d59b-466e-8590-2210e73d3a1e-istiod-ca-cert\") pod \"d5833fd5-d59b-466e-8590-2210e73d3a1e\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " Sep 11 15:22:53.553581 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.553416 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-data\") pod \"d5833fd5-d59b-466e-8590-2210e73d3a1e\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " Sep 11 15:22:53.553581 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.553444 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-workload-certs\") pod \"d5833fd5-d59b-466e-8590-2210e73d3a1e\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " Sep 11 15:22:53.553581 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.553491 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfn6v\" (UniqueName: \"kubernetes.io/projected/d5833fd5-d59b-466e-8590-2210e73d3a1e-kube-api-access-gfn6v\") pod \"d5833fd5-d59b-466e-8590-2210e73d3a1e\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " Sep 11 15:22:53.553581 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.553525 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-podinfo\") pod \"d5833fd5-d59b-466e-8590-2210e73d3a1e\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " Sep 11 15:22:53.553581 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.553562 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-token\") pod \"d5833fd5-d59b-466e-8590-2210e73d3a1e\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " Sep 11 15:22:53.553920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.553612 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5833fd5-d59b-466e-8590-2210e73d3a1e" (UID: "d5833fd5-d59b-466e-8590-2210e73d3a1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:22:53.553920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.553759 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5833fd5-d59b-466e-8590-2210e73d3a1e" (UID: "d5833fd5-d59b-466e-8590-2210e73d3a1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:22:53.553920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.553765 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5833fd5-d59b-466e-8590-2210e73d3a1e" (UID: "d5833fd5-d59b-466e-8590-2210e73d3a1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:22:53.553920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.553819 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5833fd5-d59b-466e-8590-2210e73d3a1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5833fd5-d59b-466e-8590-2210e73d3a1e" (UID: "d5833fd5-d59b-466e-8590-2210e73d3a1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:22:53.553920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.553853 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-workload-socket\") pod \"d5833fd5-d59b-466e-8590-2210e73d3a1e\" (UID: \"d5833fd5-d59b-466e-8590-2210e73d3a1e\") " Sep 11 15:22:53.554174 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.554011 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5833fd5-d59b-466e-8590-2210e73d3a1e" (UID: "d5833fd5-d59b-466e-8590-2210e73d3a1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:22:53.554297 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.554233 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:53.554297 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.554260 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5833fd5-d59b-466e-8590-2210e73d3a1e-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:53.554297 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.554277 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:53.554297 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.554291 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:53.554297 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.554304 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:53.556014 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.555991 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5833fd5-d59b-466e-8590-2210e73d3a1e" (UID: "d5833fd5-d59b-466e-8590-2210e73d3a1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:22:53.556100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.555991 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5833fd5-d59b-466e-8590-2210e73d3a1e" (UID: "d5833fd5-d59b-466e-8590-2210e73d3a1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:22:53.556191 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.556175 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5833fd5-d59b-466e-8590-2210e73d3a1e" (UID: "d5833fd5-d59b-466e-8590-2210e73d3a1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:22:53.556246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.556197 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5833fd5-d59b-466e-8590-2210e73d3a1e-kube-api-access-gfn6v" (OuterVolumeSpecName: "kube-api-access-gfn6v") pod "d5833fd5-d59b-466e-8590-2210e73d3a1e" (UID: "d5833fd5-d59b-466e-8590-2210e73d3a1e"). InnerVolumeSpecName "kube-api-access-gfn6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:22:53.655139 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.655103 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:53.655139 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.655133 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfn6v\" (UniqueName: \"kubernetes.io/projected/d5833fd5-d59b-466e-8590-2210e73d3a1e-kube-api-access-gfn6v\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:53.655139 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.655147 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:53.655383 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.655158 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5833fd5-d59b-466e-8590-2210e73d3a1e-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:22:53.881983 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:53.881879 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:54.481411 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:54.481380 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k" Sep 11 15:22:54.517983 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:54.517951 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k"] Sep 11 15:22:54.522791 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:54.522761 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-73b24789-gw-openshift-default-5b84fbf64-sc95k"] Sep 11 15:22:54.960417 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:54.960379 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:55.623605 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.623572 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz"] Sep 11 15:22:55.624182 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.624161 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerName="istio-proxy" Sep 11 15:22:55.624342 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.624204 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerName="istio-proxy" Sep 11 15:22:55.624342 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.624226 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerName="istio-proxy" Sep 11 15:22:55.624342 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.624235 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerName="istio-proxy" Sep 11 15:22:55.624510 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.624360 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerName="istio-proxy" Sep 11 15:22:55.629232 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.629207 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:55.631563 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.631540 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8c097d73-serving-cert\"" Sep 11 15:22:55.638132 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.637516 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz"] Sep 11 15:22:55.653345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.653320 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:55.671579 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.671552 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8987fde9-2dff-4646-a293-048d4a5446a2-openshift-service-ca\") pod \"maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:55.671696 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.671581 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-metrics-certs\") pod \"maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:55.671696 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.671606 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhwp\" (UniqueName: \"kubernetes.io/projected/8987fde9-2dff-4646-a293-048d4a5446a2-kube-api-access-kxhwp\") pod \"maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:55.671840 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.671727 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-maas-api-tls\") pod \"maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:55.753645 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.753602 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:55.772835 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.772810 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-maas-api-tls\") pod \"maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:55.772935 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.772838 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8987fde9-2dff-4646-a293-048d4a5446a2-openshift-service-ca\") pod \"maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:55.772935 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.772857 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-metrics-certs\") pod \"maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:55.773049 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:22:55.772956 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-serving-cert: secret "maas-api-e2e-ait-8c097d73-serving-cert" not found Sep 11 15:22:55.773049 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.772999 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhwp\" (UniqueName: \"kubernetes.io/projected/8987fde9-2dff-4646-a293-048d4a5446a2-kube-api-access-kxhwp\") pod \"maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:55.773049 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:22:55.773022 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-maas-api-tls podName:8987fde9-2dff-4646-a293-048d4a5446a2 nodeName:}" failed. No retries permitted until 2026-09-11 15:22:56.273005593 +0000 UTC m=+870.504930018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-maas-api-tls") pod "maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" (UID: "8987fde9-2dff-4646-a293-048d4a5446a2") : secret "maas-api-e2e-ait-8c097d73-serving-cert" not found Sep 11 15:22:55.773581 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.773564 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8987fde9-2dff-4646-a293-048d4a5446a2-openshift-service-ca\") pod \"maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:55.775303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.775273 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-metrics-certs\") pod \"maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:55.783920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.783899 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhwp\" (UniqueName: \"kubernetes.io/projected/8987fde9-2dff-4646-a293-048d4a5446a2-kube-api-access-kxhwp\") pod \"maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:55.910756 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:55.910715 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:56.117538 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:56.117507 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:56.218285 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:56.218192 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:22:56.266344 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:56.266313 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" path="/var/lib/kubelet/pods/d5833fd5-d59b-466e-8590-2210e73d3a1e/volumes" Sep 11 15:22:56.276497 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:56.276313 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-maas-api-tls\") pod \"maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:56.278678 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:56.278657 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-maas-api-tls\") pod \"maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:56.543304 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:56.543222 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:56.816894 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:56.816865 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz"] Sep 11 15:22:56.818795 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:22:56.818768 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8987fde9_2dff_4646_a293_048d4a5446a2.slice/crio-6634b229ebc57f9a5c619ae7a2b22a037326cb0b00e3bc5fe4bcf7c963a13c8e WatchSource:0}: Error finding container 6634b229ebc57f9a5c619ae7a2b22a037326cb0b00e3bc5fe4bcf7c963a13c8e: Status 404 returned error can't find the container with id 6634b229ebc57f9a5c619ae7a2b22a037326cb0b00e3bc5fe4bcf7c963a13c8e Sep 11 15:22:57.498825 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:57.498728 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" event={"ID":"8987fde9-2dff-4646-a293-048d4a5446a2","Type":"ContainerStarted","Data":"2e6ac6380c677325072b3ff46b7759b7deff620ded1fe73d16f029c5283e16dd"} Sep 11 15:22:57.498825 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:57.498794 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" event={"ID":"8987fde9-2dff-4646-a293-048d4a5446a2","Type":"ContainerStarted","Data":"6634b229ebc57f9a5c619ae7a2b22a037326cb0b00e3bc5fe4bcf7c963a13c8e"} Sep 11 15:22:57.499002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:57.498856 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:22:57.516100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:22:57.516054 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" podStartSLOduration=2.150589932 podStartE2EDuration="2.51604113s" totalImagesPullingTime="365.451198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:22:57.514658567 +0000 UTC m=+871.746583001" watchObservedRunningTime="2026-09-11 15:22:57.51604113 +0000 UTC m=+871.747965560" Sep 11 15:23:03.509290 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:03.509254 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:23:11.384617 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:11.384580 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:23:13.590611 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:13.590577 2558 generic.go:356] "Generic (PLEG): container finished" podID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerID="1a3b7077b97a8b1668594940a7d48574c8135e9597b531fa8835823f8e3a0446" exitCode=0 Sep 11 15:23:13.591050 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:13.590630 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" event={"ID":"77cb18c7-a74b-42ac-b515-1e8c4f5c603a","Type":"ContainerDied","Data":"1a3b7077b97a8b1668594940a7d48574c8135e9597b531fa8835823f8e3a0446"} Sep 11 15:23:13.591050 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:13.591032 2558 scope.go:117] "RemoveContainer" containerID="1a3b7077b97a8b1668594940a7d48574c8135e9597b531fa8835823f8e3a0446" Sep 11 15:23:13.592122 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:13.592086 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:23:13.592217 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:13.592165 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:23:13.592217 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:13.592197 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:23:14.603399 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:14.603366 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" event={"ID":"77cb18c7-a74b-42ac-b515-1e8c4f5c603a","Type":"ContainerStarted","Data":"f377b6e1b8e64668b454e18e4acb95064ebe223537d757451f97756525a17a73"} Sep 11 15:23:15.367185 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:15.367143 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:23:15.367411 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:15.367189 2558 patch_prober.go:28] interesting pod/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 11 15:23:15.367411 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:15.367270 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" podUID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 11 15:23:16.367068 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:16.367018 2558 patch_prober.go:28] interesting pod/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 11 15:23:16.367601 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:16.367118 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" podUID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 11 15:23:17.366706 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:17.366667 2558 patch_prober.go:28] interesting pod/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 11 15:23:17.367060 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:17.367027 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" podUID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 11 15:23:17.717723 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:17.717693 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:18.367088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:18.367042 2558 patch_prober.go:28] interesting pod/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 11 15:23:18.367271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:18.367122 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" podUID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 11 15:23:18.408431 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:18.408398 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:19.097053 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:19.096972 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:19.366553 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:19.366448 2558 patch_prober.go:28] interesting pod/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 11 15:23:19.366865 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:19.366825 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" podUID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 11 15:23:19.818475 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:19.818440 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:20.369934 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:20.369902 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:23:20.370383 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:20.370189 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:23:20.370821 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:20.370805 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:23:20.604037 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:20.604003 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:22.374196 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:22.374167 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:23.325157 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:23.325118 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:24.265856 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:24.265821 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:25.202776 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:25.202723 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:26.235761 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:26.230723 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:26.914072 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:26.914034 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:28.988296 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:28.988261 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:23:29.497936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:29.497902 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:32.238167 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:32.238132 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:32.897245 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:32.897182 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:32.970014 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:32.969970 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:33.031489 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:33.031463 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:33.092210 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:33.092179 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:33.156186 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:33.156111 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:33.455414 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:33.455322 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:33.554928 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:33.554894 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:34.446592 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:34.446558 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:35.141273 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:35.141226 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:35.789239 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:35.789199 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz"] Sep 11 15:23:35.789568 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:35.789519 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" podUID="8987fde9-2dff-4646-a293-048d4a5446a2" containerName="maas-api" containerID="cri-o://2e6ac6380c677325072b3ff46b7759b7deff620ded1fe73d16f029c5283e16dd" gracePeriod=30 Sep 11 15:23:35.995619 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:35.995590 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:36.040693 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.040624 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:23:36.149706 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.149668 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxhwp\" (UniqueName: \"kubernetes.io/projected/8987fde9-2dff-4646-a293-048d4a5446a2-kube-api-access-kxhwp\") pod \"8987fde9-2dff-4646-a293-048d4a5446a2\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " Sep 11 15:23:36.150196 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.149831 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-maas-api-tls\") pod \"8987fde9-2dff-4646-a293-048d4a5446a2\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " Sep 11 15:23:36.150196 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.149911 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-metrics-certs\") pod \"8987fde9-2dff-4646-a293-048d4a5446a2\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " Sep 11 15:23:36.150196 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.149947 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8987fde9-2dff-4646-a293-048d4a5446a2-openshift-service-ca\") pod \"8987fde9-2dff-4646-a293-048d4a5446a2\" (UID: \"8987fde9-2dff-4646-a293-048d4a5446a2\") " Sep 11 15:23:36.150343 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.150320 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8987fde9-2dff-4646-a293-048d4a5446a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8987fde9-2dff-4646-a293-048d4a5446a2" (UID: "8987fde9-2dff-4646-a293-048d4a5446a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:23:36.151991 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.151961 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8987fde9-2dff-4646-a293-048d4a5446a2-kube-api-access-kxhwp" (OuterVolumeSpecName: "kube-api-access-kxhwp") pod "8987fde9-2dff-4646-a293-048d4a5446a2" (UID: "8987fde9-2dff-4646-a293-048d4a5446a2"). InnerVolumeSpecName "kube-api-access-kxhwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:23:36.152109 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.152039 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8987fde9-2dff-4646-a293-048d4a5446a2" (UID: "8987fde9-2dff-4646-a293-048d4a5446a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:23:36.152109 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.152080 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8987fde9-2dff-4646-a293-048d4a5446a2" (UID: "8987fde9-2dff-4646-a293-048d4a5446a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:23:36.251245 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.251207 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxhwp\" (UniqueName: \"kubernetes.io/projected/8987fde9-2dff-4646-a293-048d4a5446a2-kube-api-access-kxhwp\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:23:36.251245 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.251240 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:23:36.251245 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.251249 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8987fde9-2dff-4646-a293-048d4a5446a2-metrics-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:23:36.251245 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.251257 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8987fde9-2dff-4646-a293-048d4a5446a2-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:23:36.305651 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.305568 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:36.638462 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.638380 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:36.724452 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.724415 2558 generic.go:356] "Generic (PLEG): container finished" podID="8987fde9-2dff-4646-a293-048d4a5446a2" containerID="2e6ac6380c677325072b3ff46b7759b7deff620ded1fe73d16f029c5283e16dd" exitCode=0 Sep 11 15:23:36.724621 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.724466 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" Sep 11 15:23:36.724621 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.724500 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" event={"ID":"8987fde9-2dff-4646-a293-048d4a5446a2","Type":"ContainerDied","Data":"2e6ac6380c677325072b3ff46b7759b7deff620ded1fe73d16f029c5283e16dd"} Sep 11 15:23:36.724621 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.724554 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz" event={"ID":"8987fde9-2dff-4646-a293-048d4a5446a2","Type":"ContainerDied","Data":"6634b229ebc57f9a5c619ae7a2b22a037326cb0b00e3bc5fe4bcf7c963a13c8e"} Sep 11 15:23:36.724621 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.724576 2558 scope.go:117] "RemoveContainer" containerID="2e6ac6380c677325072b3ff46b7759b7deff620ded1fe73d16f029c5283e16dd" Sep 11 15:23:36.734050 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.734036 2558 scope.go:117] "RemoveContainer" containerID="2e6ac6380c677325072b3ff46b7759b7deff620ded1fe73d16f029c5283e16dd" Sep 11 15:23:36.734297 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:23:36.734280 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6ac6380c677325072b3ff46b7759b7deff620ded1fe73d16f029c5283e16dd\": container with ID starting with 2e6ac6380c677325072b3ff46b7759b7deff620ded1fe73d16f029c5283e16dd not found: ID does not exist" containerID="2e6ac6380c677325072b3ff46b7759b7deff620ded1fe73d16f029c5283e16dd" Sep 11 15:23:36.734347 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.734305 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6ac6380c677325072b3ff46b7759b7deff620ded1fe73d16f029c5283e16dd"} err="failed to get container status \"2e6ac6380c677325072b3ff46b7759b7deff620ded1fe73d16f029c5283e16dd\": rpc error: code = NotFound desc = could not find container \"2e6ac6380c677325072b3ff46b7759b7deff620ded1fe73d16f029c5283e16dd\": container with ID starting with 2e6ac6380c677325072b3ff46b7759b7deff620ded1fe73d16f029c5283e16dd not found: ID does not exist" Sep 11 15:23:36.743681 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.743653 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz"] Sep 11 15:23:36.747779 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:36.747759 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c097d73-84d8fdfb5b-2qjrz"] Sep 11 15:23:37.101825 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.101791 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld"] Sep 11 15:23:37.102154 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.102122 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" podUID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerName="istio-proxy" containerID="cri-o://f377b6e1b8e64668b454e18e4acb95064ebe223537d757451f97756525a17a73" gracePeriod=30 Sep 11 15:23:37.118431 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.118380 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:37.221963 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.221923 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:37.551161 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.551130 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:37.828869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.828790 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp"] Sep 11 15:23:37.829345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.829330 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8987fde9-2dff-4646-a293-048d4a5446a2" containerName="maas-api" Sep 11 15:23:37.829398 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.829348 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8987fde9-2dff-4646-a293-048d4a5446a2" containerName="maas-api" Sep 11 15:23:37.829442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.829434 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8987fde9-2dff-4646-a293-048d4a5446a2" containerName="maas-api" Sep 11 15:23:37.829483 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.829449 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5833fd5-d59b-466e-8590-2210e73d3a1e" containerName="istio-proxy" Sep 11 15:23:37.834225 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.834202 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.836781 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.836720 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-df3af038-openshift-default-dockercfg-7jvnb\"" Sep 11 15:23:37.844024 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.843999 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp"] Sep 11 15:23:37.859946 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.859920 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:37.867161 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.867135 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-token\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.867278 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.867174 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k7kz\" (UniqueName: \"kubernetes.io/projected/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-kube-api-access-2k7kz\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.867278 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.867244 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-podinfo\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.867369 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.867283 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-workload-socket\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.867369 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.867308 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-data\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.867443 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.867387 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-envoy\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.867480 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.867466 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-credential-socket\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.867526 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.867486 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-workload-certs\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.867526 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.867517 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istiod-ca-cert\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.968841 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.968800 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-envoy\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.969021 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.968902 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-credential-socket\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.969021 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.968922 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-workload-certs\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.969021 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.968954 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istiod-ca-cert\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.969187 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.969085 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-token\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.969187 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.969119 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k7kz\" (UniqueName: \"kubernetes.io/projected/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-kube-api-access-2k7kz\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.969187 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.969176 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-podinfo\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.969338 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.969219 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-workload-socket\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.969338 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.969244 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-data\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.969338 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.969306 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-workload-certs\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.969686 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.969641 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-data\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.969821 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.969704 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istiod-ca-cert\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.969821 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.969716 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-workload-socket\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.969821 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.969728 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-credential-socket\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.971962 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.971920 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-envoy\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.972626 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.972600 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-podinfo\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.979357 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.979331 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k7kz\" (UniqueName: \"kubernetes.io/projected/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-kube-api-access-2k7kz\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:37.979722 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:37.979696 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-token\") pod \"e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:38.000162 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:38.000138 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:38.105591 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:38.105512 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:38.146301 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:38.146260 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:38.264867 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:38.264837 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8987fde9-2dff-4646-a293-048d4a5446a2" path="/var/lib/kubelet/pods/8987fde9-2dff-4646-a293-048d4a5446a2/volumes" Sep 11 15:23:38.437854 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:38.437694 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp"] Sep 11 15:23:38.440223 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:23:38.440192 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b723469_a9c3_40ae_b607_8fd6fa7a1e44.slice/crio-de7cb19030f7bb2481f7bd95b8c75fed388b258618a94704c0efb378a7af44d3 WatchSource:0}: Error finding container de7cb19030f7bb2481f7bd95b8c75fed388b258618a94704c0efb378a7af44d3: Status 404 returned error can't find the container with id de7cb19030f7bb2481f7bd95b8c75fed388b258618a94704c0efb378a7af44d3 Sep 11 15:23:38.442762 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:38.442713 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:23:38.442841 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:38.442811 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:23:38.442880 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:38.442846 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:23:38.533927 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:38.533901 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:38.739164 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:38.739077 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" event={"ID":"2b723469-a9c3-40ae-b607-8fd6fa7a1e44","Type":"ContainerStarted","Data":"132578e3ef86d1feb83a5809593a652a5e67687c079af0791a5e0ca4ebdc67f0"} Sep 11 15:23:38.739164 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:38.739142 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" event={"ID":"2b723469-a9c3-40ae-b607-8fd6fa7a1e44","Type":"ContainerStarted","Data":"de7cb19030f7bb2481f7bd95b8c75fed388b258618a94704c0efb378a7af44d3"} Sep 11 15:23:38.757725 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:38.757652 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" podStartSLOduration=1.757633136 podStartE2EDuration="1.757633136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:23:38.756838639 +0000 UTC m=+912.988763069" watchObservedRunningTime="2026-09-11 15:23:38.757633136 +0000 UTC m=+912.989557568" Sep 11 15:23:39.146786 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:39.146757 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:39.151902 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:39.151878 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:39.404351 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:39.404267 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:39.745007 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:39.744930 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:39.745956 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:39.745934 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:40.118695 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:40.118620 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:40.216147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:40.216115 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:41.231095 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:41.231063 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:41.889853 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:41.889818 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc"] Sep 11 15:23:41.894167 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:41.894143 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:41.896604 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:41.896579 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-df3af038-serving-cert\"" Sep 11 15:23:41.902194 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:41.902171 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc"] Sep 11 15:23:41.921441 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:41.921416 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:42.008001 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.007962 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-maas-api-tls\") pod \"maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:42.008428 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.008404 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5rr\" (UniqueName: \"kubernetes.io/projected/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-kube-api-access-nj5rr\") pod \"maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:42.008586 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.008567 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-metrics-certs\") pod \"maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:42.008758 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.008725 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-openshift-service-ca\") pod \"maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:42.022521 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.022481 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:42.110163 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.110128 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5rr\" (UniqueName: \"kubernetes.io/projected/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-kube-api-access-nj5rr\") pod \"maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:42.110163 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.110169 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-metrics-certs\") pod \"maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:42.110392 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.110261 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-openshift-service-ca\") pod \"maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:42.110392 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.110283 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-maas-api-tls\") pod \"maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:42.110514 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:23:42.110401 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-serving-cert: secret "maas-api-e2e-derive-df3af038-serving-cert" not found Sep 11 15:23:42.110514 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:23:42.110471 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-maas-api-tls podName:b1a7b077-9e8f-46a0-bb67-036f3c3a2e32 nodeName:}" failed. No retries permitted until 2026-09-11 15:23:42.610449912 +0000 UTC m=+916.842374327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-maas-api-tls") pod "maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" (UID: "b1a7b077-9e8f-46a0-bb67-036f3c3a2e32") : secret "maas-api-e2e-derive-df3af038-serving-cert" not found Sep 11 15:23:42.111129 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.111102 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-openshift-service-ca\") pod \"maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:42.112889 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.112870 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-metrics-certs\") pod \"maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:42.119506 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.119483 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5rr\" (UniqueName: \"kubernetes.io/projected/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-kube-api-access-nj5rr\") pod \"maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:42.370470 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.370440 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:23:42.386045 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.386020 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:42.412999 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.412969 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2fng\" (UniqueName: \"kubernetes.io/projected/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-kube-api-access-w2fng\") pod \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " Sep 11 15:23:42.413155 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.413022 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-credential-socket\") pod \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " Sep 11 15:23:42.413217 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.413188 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-podinfo\") pod \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " Sep 11 15:23:42.413332 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.413310 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "77cb18c7-a74b-42ac-b515-1e8c4f5c603a" (UID: "77cb18c7-a74b-42ac-b515-1e8c4f5c603a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:23:42.413458 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.413439 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istiod-ca-cert\") pod \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " Sep 11 15:23:42.413548 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.413536 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-workload-socket\") pod \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " Sep 11 15:23:42.413600 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.413568 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-token\") pod \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " Sep 11 15:23:42.413661 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.413606 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-data\") pod \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " Sep 11 15:23:42.413661 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.413630 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-workload-certs\") pod \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " Sep 11 15:23:42.414151 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.413974 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-data" (OuterVolumeSpecName: "istio-data") pod "77cb18c7-a74b-42ac-b515-1e8c4f5c603a" (UID: "77cb18c7-a74b-42ac-b515-1e8c4f5c603a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:23:42.414151 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.414041 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:23:42.414151 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.414104 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "77cb18c7-a74b-42ac-b515-1e8c4f5c603a" (UID: "77cb18c7-a74b-42ac-b515-1e8c4f5c603a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:23:42.414395 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.414376 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "77cb18c7-a74b-42ac-b515-1e8c4f5c603a" (UID: "77cb18c7-a74b-42ac-b515-1e8c4f5c603a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:23:42.414395 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.414386 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "77cb18c7-a74b-42ac-b515-1e8c4f5c603a" (UID: "77cb18c7-a74b-42ac-b515-1e8c4f5c603a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:23:42.415449 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.415421 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "77cb18c7-a74b-42ac-b515-1e8c4f5c603a" (UID: "77cb18c7-a74b-42ac-b515-1e8c4f5c603a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:23:42.415656 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.415633 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-kube-api-access-w2fng" (OuterVolumeSpecName: "kube-api-access-w2fng") pod "77cb18c7-a74b-42ac-b515-1e8c4f5c603a" (UID: "77cb18c7-a74b-42ac-b515-1e8c4f5c603a"). InnerVolumeSpecName "kube-api-access-w2fng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:23:42.416230 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.416204 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-token" (OuterVolumeSpecName: "istio-token") pod "77cb18c7-a74b-42ac-b515-1e8c4f5c603a" (UID: "77cb18c7-a74b-42ac-b515-1e8c4f5c603a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:23:42.491896 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.491815 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:42.515183 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.515159 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-envoy\") pod \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\" (UID: \"77cb18c7-a74b-42ac-b515-1e8c4f5c603a\") " Sep 11 15:23:42.515433 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.515417 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2fng\" (UniqueName: \"kubernetes.io/projected/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-kube-api-access-w2fng\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:23:42.515519 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.515439 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:23:42.515519 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.515455 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:23:42.515519 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.515469 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:23:42.515519 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.515482 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:23:42.515519 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.515495 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:23:42.515519 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.515507 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:23:42.517146 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.517126 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "77cb18c7-a74b-42ac-b515-1e8c4f5c603a" (UID: "77cb18c7-a74b-42ac-b515-1e8c4f5c603a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:23:42.585492 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.585455 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:42.616885 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.616839 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-maas-api-tls\") pod \"maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:42.617047 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.617032 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cb18c7-a74b-42ac-b515-1e8c4f5c603a-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:23:42.621089 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.621017 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-maas-api-tls\") pod \"maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:42.762589 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.762497 2558 generic.go:356] "Generic (PLEG): container finished" podID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerID="f377b6e1b8e64668b454e18e4acb95064ebe223537d757451f97756525a17a73" exitCode=0 Sep 11 15:23:42.762589 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.762550 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" Sep 11 15:23:42.762792 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.762586 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" event={"ID":"77cb18c7-a74b-42ac-b515-1e8c4f5c603a","Type":"ContainerDied","Data":"f377b6e1b8e64668b454e18e4acb95064ebe223537d757451f97756525a17a73"} Sep 11 15:23:42.762792 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.762642 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld" event={"ID":"77cb18c7-a74b-42ac-b515-1e8c4f5c603a","Type":"ContainerDied","Data":"d641459e2e127c5cba22eccf3b106280d9d86dbd5ce065f16b2e136395088918"} Sep 11 15:23:42.762792 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.762670 2558 scope.go:117] "RemoveContainer" containerID="f377b6e1b8e64668b454e18e4acb95064ebe223537d757451f97756525a17a73" Sep 11 15:23:42.772600 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.772581 2558 scope.go:117] "RemoveContainer" containerID="1a3b7077b97a8b1668594940a7d48574c8135e9597b531fa8835823f8e3a0446" Sep 11 15:23:42.780844 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.780822 2558 scope.go:117] "RemoveContainer" containerID="f377b6e1b8e64668b454e18e4acb95064ebe223537d757451f97756525a17a73" Sep 11 15:23:42.781139 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:23:42.781116 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f377b6e1b8e64668b454e18e4acb95064ebe223537d757451f97756525a17a73\": container with ID starting with f377b6e1b8e64668b454e18e4acb95064ebe223537d757451f97756525a17a73 not found: ID does not exist" containerID="f377b6e1b8e64668b454e18e4acb95064ebe223537d757451f97756525a17a73" Sep 11 15:23:42.781186 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.781147 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f377b6e1b8e64668b454e18e4acb95064ebe223537d757451f97756525a17a73"} err="failed to get container status \"f377b6e1b8e64668b454e18e4acb95064ebe223537d757451f97756525a17a73\": rpc error: code = NotFound desc = could not find container \"f377b6e1b8e64668b454e18e4acb95064ebe223537d757451f97756525a17a73\": container with ID starting with f377b6e1b8e64668b454e18e4acb95064ebe223537d757451f97756525a17a73 not found: ID does not exist" Sep 11 15:23:42.781186 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.781164 2558 scope.go:117] "RemoveContainer" containerID="1a3b7077b97a8b1668594940a7d48574c8135e9597b531fa8835823f8e3a0446" Sep 11 15:23:42.781420 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:23:42.781402 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3b7077b97a8b1668594940a7d48574c8135e9597b531fa8835823f8e3a0446\": container with ID starting with 1a3b7077b97a8b1668594940a7d48574c8135e9597b531fa8835823f8e3a0446 not found: ID does not exist" containerID="1a3b7077b97a8b1668594940a7d48574c8135e9597b531fa8835823f8e3a0446" Sep 11 15:23:42.781475 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.781426 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3b7077b97a8b1668594940a7d48574c8135e9597b531fa8835823f8e3a0446"} err="failed to get container status \"1a3b7077b97a8b1668594940a7d48574c8135e9597b531fa8835823f8e3a0446\": rpc error: code = NotFound desc = could not find container \"1a3b7077b97a8b1668594940a7d48574c8135e9597b531fa8835823f8e3a0446\": container with ID starting with 1a3b7077b97a8b1668594940a7d48574c8135e9597b531fa8835823f8e3a0446 not found: ID does not exist" Sep 11 15:23:42.784820 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.784798 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld"] Sep 11 15:23:42.788871 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.788848 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8c097d73-openshift-default-7b7464c654-vgtld"] Sep 11 15:23:42.807477 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:42.807451 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:43.082971 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:43.082943 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc"] Sep 11 15:23:43.084409 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:23:43.084385 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a7b077_9e8f_46a0_bb67_036f3c3a2e32.slice/crio-fc57c92bdede0097213b3175ea3141f5423fff1b2bf86bdd98c617924ae33a53 WatchSource:0}: Error finding container fc57c92bdede0097213b3175ea3141f5423fff1b2bf86bdd98c617924ae33a53: Status 404 returned error can't find the container with id fc57c92bdede0097213b3175ea3141f5423fff1b2bf86bdd98c617924ae33a53 Sep 11 15:23:43.769243 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:43.769151 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" event={"ID":"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32","Type":"ContainerStarted","Data":"4a74288f9fef7b7ee57b208f99fce015ed9f625c95d638c011b216d8fc5a3c19"} Sep 11 15:23:43.769243 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:43.769213 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:43.769793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:43.769278 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" event={"ID":"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32","Type":"ContainerStarted","Data":"fc57c92bdede0097213b3175ea3141f5423fff1b2bf86bdd98c617924ae33a53"} Sep 11 15:23:43.787846 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:43.787798 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" podStartSLOduration=2.393072595 podStartE2EDuration="2.787785098s" totalImagesPullingTime="394.712503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:23:43.785864996 +0000 UTC m=+918.017789440" watchObservedRunningTime="2026-09-11 15:23:43.787785098 +0000 UTC m=+918.019709530" Sep 11 15:23:44.263927 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:44.263893 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" path="/var/lib/kubelet/pods/77cb18c7-a74b-42ac-b515-1e8c4f5c603a/volumes" Sep 11 15:23:46.287527 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:46.287490 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:23:49.781226 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:49.781197 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:23:55.785984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:55.785937 2558 patch_prober.go:28] interesting pod/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 15:23:55.786394 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:55.786002 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 15:23:56.846364 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:56.846267 2558 generic.go:356] "Generic (PLEG): container finished" podID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerID="132578e3ef86d1feb83a5809593a652a5e67687c079af0791a5e0ca4ebdc67f0" exitCode=0 Sep 11 15:23:56.846364 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:56.846323 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" event={"ID":"2b723469-a9c3-40ae-b607-8fd6fa7a1e44","Type":"ContainerDied","Data":"132578e3ef86d1feb83a5809593a652a5e67687c079af0791a5e0ca4ebdc67f0"} Sep 11 15:23:56.846823 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:56.846715 2558 scope.go:117] "RemoveContainer" containerID="132578e3ef86d1feb83a5809593a652a5e67687c079af0791a5e0ca4ebdc67f0" Sep 11 15:23:56.847933 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:56.847890 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:23:56.848034 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:56.847993 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:23:56.848074 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:56.848039 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:23:56.880593 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:56.880524 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:23:57.854487 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:57.854442 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" event={"ID":"2b723469-a9c3-40ae-b607-8fd6fa7a1e44","Type":"ContainerStarted","Data":"1b04af8b4472b3a603615432e587450c36d4c88a7e273aab9f4e9f98d86d90ca"} Sep 11 15:23:58.147319 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:58.147289 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:23:58.147506 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:58.147485 2558 patch_prober.go:28] interesting pod/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 11 15:23:58.147552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:58.147535 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 11 15:23:59.147443 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:59.147405 2558 patch_prober.go:28] interesting pod/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 11 15:23:59.147867 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:23:59.147461 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 11 15:24:00.147188 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:00.147151 2558 patch_prober.go:28] interesting pod/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 11 15:24:00.147375 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:00.147212 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 11 15:24:00.384205 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:00.384171 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:24:01.147941 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:01.147906 2558 patch_prober.go:28] interesting pod/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 11 15:24:01.148124 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:01.147961 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 11 15:24:02.146963 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:02.146926 2558 patch_prober.go:28] interesting pod/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 11 15:24:02.147428 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:02.146982 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 11 15:24:03.292919 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:03.292887 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:24:03.293385 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:03.293289 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:24:03.294015 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:03.293994 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:24:03.981314 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:03.981279 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:05.328957 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:05.328925 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:05.509660 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:05.509627 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:06.311453 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:06.311416 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:07.023492 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:07.023461 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:07.753367 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:07.753322 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:11.725620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:11.725589 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:12.274493 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.274461 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc"] Sep 11 15:24:12.274731 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.274704 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" podUID="b1a7b077-9e8f-46a0-bb67-036f3c3a2e32" containerName="maas-api" containerID="cri-o://4a74288f9fef7b7ee57b208f99fce015ed9f625c95d638c011b216d8fc5a3c19" gracePeriod=30 Sep 11 15:24:12.302988 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.302957 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:12.403358 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.403329 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:12.491088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.491055 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:12.526048 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.525990 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:24:12.592654 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.592622 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-metrics-certs\") pod \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " Sep 11 15:24:12.592835 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.592701 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj5rr\" (UniqueName: \"kubernetes.io/projected/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-kube-api-access-nj5rr\") pod \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " Sep 11 15:24:12.592835 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.592729 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-openshift-service-ca\") pod \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " Sep 11 15:24:12.592835 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.592797 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-maas-api-tls\") pod \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\" (UID: \"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32\") " Sep 11 15:24:12.593305 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.593270 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1a7b077-9e8f-46a0-bb67-036f3c3a2e32" (UID: "b1a7b077-9e8f-46a0-bb67-036f3c3a2e32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:24:12.596242 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.596208 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1a7b077-9e8f-46a0-bb67-036f3c3a2e32" (UID: "b1a7b077-9e8f-46a0-bb67-036f3c3a2e32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:24:12.596372 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.596281 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b1a7b077-9e8f-46a0-bb67-036f3c3a2e32" (UID: "b1a7b077-9e8f-46a0-bb67-036f3c3a2e32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:24:12.596372 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.596354 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-kube-api-access-nj5rr" (OuterVolumeSpecName: "kube-api-access-nj5rr") pod "b1a7b077-9e8f-46a0-bb67-036f3c3a2e32" (UID: "b1a7b077-9e8f-46a0-bb67-036f3c3a2e32"). InnerVolumeSpecName "kube-api-access-nj5rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:24:12.600917 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.600877 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:12.694169 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.694126 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:12.694169 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.694166 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-metrics-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:12.694353 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.694182 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj5rr\" (UniqueName: \"kubernetes.io/projected/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-kube-api-access-nj5rr\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:12.694353 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.694197 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:12.743120 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.743086 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp"] Sep 11 15:24:12.743461 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.743307 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerName="istio-proxy" containerID="cri-o://1b04af8b4472b3a603615432e587450c36d4c88a7e273aab9f4e9f98d86d90ca" gracePeriod=30 Sep 11 15:24:12.746750 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.746709 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:12.949339 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.949308 2558 generic.go:356] "Generic (PLEG): container finished" podID="b1a7b077-9e8f-46a0-bb67-036f3c3a2e32" containerID="4a74288f9fef7b7ee57b208f99fce015ed9f625c95d638c011b216d8fc5a3c19" exitCode=0 Sep 11 15:24:12.949511 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.949352 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" Sep 11 15:24:12.949511 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.949401 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" event={"ID":"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32","Type":"ContainerDied","Data":"4a74288f9fef7b7ee57b208f99fce015ed9f625c95d638c011b216d8fc5a3c19"} Sep 11 15:24:12.949511 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.949452 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc" event={"ID":"b1a7b077-9e8f-46a0-bb67-036f3c3a2e32","Type":"ContainerDied","Data":"fc57c92bdede0097213b3175ea3141f5423fff1b2bf86bdd98c617924ae33a53"} Sep 11 15:24:12.949511 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.949475 2558 scope.go:117] "RemoveContainer" containerID="4a74288f9fef7b7ee57b208f99fce015ed9f625c95d638c011b216d8fc5a3c19" Sep 11 15:24:12.959518 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.959501 2558 scope.go:117] "RemoveContainer" containerID="4a74288f9fef7b7ee57b208f99fce015ed9f625c95d638c011b216d8fc5a3c19" Sep 11 15:24:12.959796 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:24:12.959772 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a74288f9fef7b7ee57b208f99fce015ed9f625c95d638c011b216d8fc5a3c19\": container with ID starting with 4a74288f9fef7b7ee57b208f99fce015ed9f625c95d638c011b216d8fc5a3c19 not found: ID does not exist" containerID="4a74288f9fef7b7ee57b208f99fce015ed9f625c95d638c011b216d8fc5a3c19" Sep 11 15:24:12.959854 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.959806 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a74288f9fef7b7ee57b208f99fce015ed9f625c95d638c011b216d8fc5a3c19"} err="failed to get container status \"4a74288f9fef7b7ee57b208f99fce015ed9f625c95d638c011b216d8fc5a3c19\": rpc error: code = NotFound desc = could not find container \"4a74288f9fef7b7ee57b208f99fce015ed9f625c95d638c011b216d8fc5a3c19\": container with ID starting with 4a74288f9fef7b7ee57b208f99fce015ed9f625c95d638c011b216d8fc5a3c19 not found: ID does not exist" Sep 11 15:24:12.973806 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.973785 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc"] Sep 11 15:24:12.976549 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:12.976528 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-df3af038-9f7fbb9b5-22bmc"] Sep 11 15:24:13.123287 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:13.123258 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:13.358915 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:13.358837 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:13.457657 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:13.457629 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:13.558004 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:13.557969 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:13.940621 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:13.940591 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:14.264434 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:14.264355 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1a7b077-9e8f-46a0-bb67-036f3c3a2e32" path="/var/lib/kubelet/pods/b1a7b077-9e8f-46a0-bb67-036f3c3a2e32/volumes" Sep 11 15:24:14.702418 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:14.702390 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:14.803643 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:14.803602 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:14.932423 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:14.932386 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:15.840352 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:15.840320 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:17.195668 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:17.192639 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:24:17.980692 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:17.980666 2558 generic.go:356] "Generic (PLEG): container finished" podID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerID="1b04af8b4472b3a603615432e587450c36d4c88a7e273aab9f4e9f98d86d90ca" exitCode=0 Sep 11 15:24:17.980856 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:17.980699 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" event={"ID":"2b723469-a9c3-40ae-b607-8fd6fa7a1e44","Type":"ContainerDied","Data":"1b04af8b4472b3a603615432e587450c36d4c88a7e273aab9f4e9f98d86d90ca"} Sep 11 15:24:17.980856 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:17.980793 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" event={"ID":"2b723469-a9c3-40ae-b607-8fd6fa7a1e44","Type":"ContainerDied","Data":"de7cb19030f7bb2481f7bd95b8c75fed388b258618a94704c0efb378a7af44d3"} Sep 11 15:24:17.980856 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:17.980823 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de7cb19030f7bb2481f7bd95b8c75fed388b258618a94704c0efb378a7af44d3" Sep 11 15:24:17.980856 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:17.980842 2558 scope.go:117] "RemoveContainer" containerID="132578e3ef86d1feb83a5809593a652a5e67687c079af0791a5e0ca4ebdc67f0" Sep 11 15:24:17.990546 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:17.990439 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:24:18.038713 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.038657 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-credential-socket\") pod \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " Sep 11 15:24:18.038713 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.038701 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-data\") pod \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " Sep 11 15:24:18.038916 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.038769 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-podinfo\") pod \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " Sep 11 15:24:18.038916 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.038792 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-token\") pod \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " Sep 11 15:24:18.038916 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.038824 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k7kz\" (UniqueName: \"kubernetes.io/projected/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-kube-api-access-2k7kz\") pod \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " Sep 11 15:24:18.039054 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.038923 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2b723469-a9c3-40ae-b607-8fd6fa7a1e44" (UID: "2b723469-a9c3-40ae-b607-8fd6fa7a1e44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:18.039054 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.038950 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-workload-certs\") pod \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " Sep 11 15:24:18.039054 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.038982 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istiod-ca-cert\") pod \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " Sep 11 15:24:18.039054 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.039027 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-data" (OuterVolumeSpecName: "istio-data") pod "2b723469-a9c3-40ae-b607-8fd6fa7a1e44" (UID: "2b723469-a9c3-40ae-b607-8fd6fa7a1e44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:18.039054 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.039038 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-envoy\") pod \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " Sep 11 15:24:18.039294 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.039086 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-workload-socket\") pod \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\" (UID: \"2b723469-a9c3-40ae-b607-8fd6fa7a1e44\") " Sep 11 15:24:18.039294 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.039202 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2b723469-a9c3-40ae-b607-8fd6fa7a1e44" (UID: "2b723469-a9c3-40ae-b607-8fd6fa7a1e44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:18.039395 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.039295 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2b723469-a9c3-40ae-b607-8fd6fa7a1e44" (UID: "2b723469-a9c3-40ae-b607-8fd6fa7a1e44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:24:18.039468 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.039402 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2b723469-a9c3-40ae-b607-8fd6fa7a1e44" (UID: "2b723469-a9c3-40ae-b607-8fd6fa7a1e44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:18.039522 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.039477 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:18.039522 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.039498 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:18.039522 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.039511 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:18.039680 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.039524 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:18.041004 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.040979 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2b723469-a9c3-40ae-b607-8fd6fa7a1e44" (UID: "2b723469-a9c3-40ae-b607-8fd6fa7a1e44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:24:18.041157 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.041087 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-kube-api-access-2k7kz" (OuterVolumeSpecName: "kube-api-access-2k7kz") pod "2b723469-a9c3-40ae-b607-8fd6fa7a1e44" (UID: "2b723469-a9c3-40ae-b607-8fd6fa7a1e44"). InnerVolumeSpecName "kube-api-access-2k7kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:24:18.041157 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.041138 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-token" (OuterVolumeSpecName: "istio-token") pod "2b723469-a9c3-40ae-b607-8fd6fa7a1e44" (UID: "2b723469-a9c3-40ae-b607-8fd6fa7a1e44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:24:18.041157 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.041143 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2b723469-a9c3-40ae-b607-8fd6fa7a1e44" (UID: "2b723469-a9c3-40ae-b607-8fd6fa7a1e44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:18.140458 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.140414 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:18.140458 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.140453 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:18.140458 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.140464 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k7kz\" (UniqueName: \"kubernetes.io/projected/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-kube-api-access-2k7kz\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:18.140458 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.140474 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:18.140712 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.140483 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b723469-a9c3-40ae-b607-8fd6fa7a1e44-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:18.987255 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:18.987224 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp" Sep 11 15:24:19.020378 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:19.020349 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp"] Sep 11 15:24:19.023802 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:19.023771 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-df3af038-openshift-default-584dd9c64-r9bfp"] Sep 11 15:24:20.264285 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:20.264254 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" path="/var/lib/kubelet/pods/2b723469-a9c3-40ae-b607-8fd6fa7a1e44/volumes" Sep 11 15:24:20.298665 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:20.298631 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:20.732615 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:20.732580 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:20.836923 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:20.836895 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:20.931582 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:20.931544 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:21.030734 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:21.030654 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:21.437505 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:21.437473 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:30.588710 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:30.588676 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:30.688881 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:30.688825 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:30.828963 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:30.828931 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk"] Sep 11 15:24:30.829249 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:30.829214 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" podUID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerName="istio-proxy" containerID="cri-o://62aebcb075ebc4571a03730fd6a2ea3db38206e02fc30e2da3deff4cb0140bd8" gracePeriod=30 Sep 11 15:24:30.846708 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:30.846632 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:31.304072 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.304040 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:31.480681 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.480648 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:24:31.589160 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589081 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b"] Sep 11 15:24:31.589604 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589583 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerName="istio-proxy" Sep 11 15:24:31.589675 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589608 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerName="istio-proxy" Sep 11 15:24:31.589675 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589625 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a7b077-9e8f-46a0-bb67-036f3c3a2e32" containerName="maas-api" Sep 11 15:24:31.589675 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589634 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a7b077-9e8f-46a0-bb67-036f3c3a2e32" containerName="maas-api" Sep 11 15:24:31.589675 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589653 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerName="istio-proxy" Sep 11 15:24:31.589675 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589662 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerName="istio-proxy" Sep 11 15:24:31.589675 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589674 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerName="istio-proxy" Sep 11 15:24:31.589984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589683 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerName="istio-proxy" Sep 11 15:24:31.589984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589795 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerName="istio-proxy" Sep 11 15:24:31.589984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589806 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerName="istio-proxy" Sep 11 15:24:31.589984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589818 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="77cb18c7-a74b-42ac-b515-1e8c4f5c603a" containerName="istio-proxy" Sep 11 15:24:31.589984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589826 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a7b077-9e8f-46a0-bb67-036f3c3a2e32" containerName="maas-api" Sep 11 15:24:31.589984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.589837 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerName="istio-proxy" Sep 11 15:24:31.594472 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.594453 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.596577 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.596557 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-ca78ef-openshift-default-dockercfg-ngtc2\"" Sep 11 15:24:31.604041 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.604017 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b"] Sep 11 15:24:31.615537 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.615512 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:31.655169 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.655141 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-credential-socket\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.655286 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.655197 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-podinfo\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.655286 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.655248 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istiod-ca-cert\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.655286 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.655272 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-token\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.655415 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.655325 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgc8t\" (UniqueName: \"kubernetes.io/projected/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-kube-api-access-pgc8t\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.655415 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.655358 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-data\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.655415 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.655397 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-workload-socket\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.655528 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.655430 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-workload-certs\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.655528 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.655460 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-envoy\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.755915 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.755877 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-workload-certs\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.755915 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.755915 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-envoy\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.756177 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.755942 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-credential-socket\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.756177 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.755974 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-podinfo\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.756177 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.755993 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istiod-ca-cert\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.756177 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.756014 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-token\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.756177 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.756040 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgc8t\" (UniqueName: \"kubernetes.io/projected/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-kube-api-access-pgc8t\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.756448 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.756211 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-data\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.756448 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.756289 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-workload-socket\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.756448 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.756344 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-workload-certs\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.756605 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.756582 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-credential-socket\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.756662 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.756601 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-data\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.756662 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.756631 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-workload-socket\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.756962 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.756942 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istiod-ca-cert\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.758473 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.758449 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-podinfo\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.758734 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.758714 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-envoy\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.763787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.763763 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-token\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.763892 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.763872 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgc8t\" (UniqueName: \"kubernetes.io/projected/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-kube-api-access-pgc8t\") pod \"e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:31.908279 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:31.908249 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:32.011380 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:32.011352 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:32.184114 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:32.184021 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b"] Sep 11 15:24:32.187083 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:24:32.187053 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad748dcb_f098_4f4d_8bc9_bdbccf34bd46.slice/crio-fed149992f95ac333b2a22d03f7533d5f8027d82efa31e653a9b7abe6c59dc6d WatchSource:0}: Error finding container fed149992f95ac333b2a22d03f7533d5f8027d82efa31e653a9b7abe6c59dc6d: Status 404 returned error can't find the container with id fed149992f95ac333b2a22d03f7533d5f8027d82efa31e653a9b7abe6c59dc6d Sep 11 15:24:32.189588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:32.189556 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:24:32.189651 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:32.189623 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:24:32.189688 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:32.189652 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:24:33.065527 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:33.065488 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" event={"ID":"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46","Type":"ContainerStarted","Data":"dede56e3668d04ebcaa55679398510bf722ba556c1c2777ae8a9e3d9de0bf81f"} Sep 11 15:24:33.065992 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:33.065574 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" event={"ID":"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46","Type":"ContainerStarted","Data":"fed149992f95ac333b2a22d03f7533d5f8027d82efa31e653a9b7abe6c59dc6d"} Sep 11 15:24:33.086873 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:33.086826 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" podStartSLOduration=2.086811185 podStartE2EDuration="2.086811185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:24:33.084049222 +0000 UTC m=+967.315973665" watchObservedRunningTime="2026-09-11 15:24:33.086811185 +0000 UTC m=+967.318735616" Sep 11 15:24:33.909362 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:33.909332 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:33.914363 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:33.914324 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:34.071964 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:34.071934 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:34.072866 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:34.072843 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:34.731553 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:34.731520 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:35.011765 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:35.011640 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:35.109544 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:35.109515 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:35.630601 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:35.630564 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:36.083617 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.083589 2558 generic.go:356] "Generic (PLEG): container finished" podID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerID="62aebcb075ebc4571a03730fd6a2ea3db38206e02fc30e2da3deff4cb0140bd8" exitCode=0 Sep 11 15:24:36.083771 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.083657 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" event={"ID":"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02","Type":"ContainerDied","Data":"62aebcb075ebc4571a03730fd6a2ea3db38206e02fc30e2da3deff4cb0140bd8"} Sep 11 15:24:36.083771 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.083705 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:24:36.083771 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.083725 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" event={"ID":"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02","Type":"ContainerDied","Data":"59071d83714fe7b5d1b3cb7175c82c70f00def7e67ad8c1187f2887cf212c6bb"} Sep 11 15:24:36.083771 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.083760 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59071d83714fe7b5d1b3cb7175c82c70f00def7e67ad8c1187f2887cf212c6bb" Sep 11 15:24:36.083942 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.083775 2558 scope.go:117] "RemoveContainer" containerID="1bb90eaebbf115f2601db0f019d751293d5641178c984dc2925e6aae302e55b8" Sep 11 15:24:36.196381 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.196341 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-credential-socket\") pod \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " Sep 11 15:24:36.196861 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.196397 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv9nh\" (UniqueName: \"kubernetes.io/projected/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-kube-api-access-rv9nh\") pod \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " Sep 11 15:24:36.196861 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.196426 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-workload-socket\") pod \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " Sep 11 15:24:36.196861 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.196450 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-workload-certs\") pod \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " Sep 11 15:24:36.196861 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.196624 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-podinfo\") pod \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " Sep 11 15:24:36.196861 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.196681 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-token\") pod \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " Sep 11 15:24:36.196861 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.196708 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" (UID: "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:36.196861 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.196761 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-data\") pod \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " Sep 11 15:24:36.196861 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.196766 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" (UID: "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:36.196861 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.196793 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-envoy\") pod \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " Sep 11 15:24:36.196861 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.196820 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istiod-ca-cert\") pod \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\" (UID: \"eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02\") " Sep 11 15:24:36.197466 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.196885 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" (UID: "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:36.197466 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.197003 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-data" (OuterVolumeSpecName: "istio-data") pod "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" (UID: "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:36.197466 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.197181 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" (UID: "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:24:36.197466 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.197452 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:36.197690 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.197566 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:36.197690 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.197589 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:36.197690 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.197607 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:36.199063 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.199036 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" (UID: "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:24:36.199063 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.199052 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-kube-api-access-rv9nh" (OuterVolumeSpecName: "kube-api-access-rv9nh") pod "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" (UID: "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02"). InnerVolumeSpecName "kube-api-access-rv9nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:24:36.199206 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.199060 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-token" (OuterVolumeSpecName: "istio-token") pod "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" (UID: "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:24:36.199206 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.199130 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" (UID: "eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:24:36.298783 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.298726 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:36.298783 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.298785 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:36.298957 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.298801 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:36.298957 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.298814 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:36.298957 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.298828 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv9nh\" (UniqueName: \"kubernetes.io/projected/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02-kube-api-access-rv9nh\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:24:36.305509 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.305479 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg"] Sep 11 15:24:36.306019 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.306004 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerName="istio-proxy" Sep 11 15:24:36.306075 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.306022 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerName="istio-proxy" Sep 11 15:24:36.306075 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.306047 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerName="istio-proxy" Sep 11 15:24:36.306075 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.306056 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b723469-a9c3-40ae-b607-8fd6fa7a1e44" containerName="istio-proxy" Sep 11 15:24:36.306075 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.306068 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerName="istio-proxy" Sep 11 15:24:36.306198 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.306078 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerName="istio-proxy" Sep 11 15:24:36.306198 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.306171 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerName="istio-proxy" Sep 11 15:24:36.310685 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.310658 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.313153 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.313131 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-ca78ef-serving-cert\"" Sep 11 15:24:36.317849 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.317829 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg"] Sep 11 15:24:36.333635 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.333612 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:36.395594 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.395566 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:36.399957 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.399925 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6925ff-2827-4653-bf35-be7f84df2a30-openshift-service-ca\") pod \"maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.400110 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.399973 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4w94\" (UniqueName: \"kubernetes.io/projected/bb6925ff-2827-4653-bf35-be7f84df2a30-kube-api-access-m4w94\") pod \"maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.400110 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.400001 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6925ff-2827-4653-bf35-be7f84df2a30-metrics-certs\") pod \"maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.400110 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.400022 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb6925ff-2827-4653-bf35-be7f84df2a30-maas-api-tls\") pod \"maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.493012 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.492937 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:36.501153 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.501118 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6925ff-2827-4653-bf35-be7f84df2a30-openshift-service-ca\") pod \"maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.501330 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.501183 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4w94\" (UniqueName: \"kubernetes.io/projected/bb6925ff-2827-4653-bf35-be7f84df2a30-kube-api-access-m4w94\") pod \"maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.501330 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.501220 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6925ff-2827-4653-bf35-be7f84df2a30-metrics-certs\") pod \"maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.501330 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.501255 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb6925ff-2827-4653-bf35-be7f84df2a30-maas-api-tls\") pod \"maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.502001 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.501952 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6925ff-2827-4653-bf35-be7f84df2a30-openshift-service-ca\") pod \"maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.504661 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.504630 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb6925ff-2827-4653-bf35-be7f84df2a30-maas-api-tls\") pod \"maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.504793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.504712 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6925ff-2827-4653-bf35-be7f84df2a30-metrics-certs\") pod \"maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.509724 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.509699 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4w94\" (UniqueName: \"kubernetes.io/projected/bb6925ff-2827-4653-bf35-be7f84df2a30-kube-api-access-m4w94\") pod \"maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.623540 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.623504 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:36.815324 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.815239 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:36.893580 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.893555 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg"] Sep 11 15:24:36.895215 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:24:36.895175 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6925ff_2827_4653_bf35_be7f84df2a30.slice/crio-7ca6d7dfd0ed1c528c2cfe0d5ba59f2b0b01778ea0ca703f3394f932a14a4678 WatchSource:0}: Error finding container 7ca6d7dfd0ed1c528c2cfe0d5ba59f2b0b01778ea0ca703f3394f932a14a4678: Status 404 returned error can't find the container with id 7ca6d7dfd0ed1c528c2cfe0d5ba59f2b0b01778ea0ca703f3394f932a14a4678 Sep 11 15:24:36.895662 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.895636 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:36.896532 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:36.896514 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 15:24:37.009154 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:37.009120 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:37.091097 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:37.091021 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk" Sep 11 15:24:37.092394 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:37.092367 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" event={"ID":"bb6925ff-2827-4653-bf35-be7f84df2a30","Type":"ContainerStarted","Data":"7ca6d7dfd0ed1c528c2cfe0d5ba59f2b0b01778ea0ca703f3394f932a14a4678"} Sep 11 15:24:37.121880 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:37.121847 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk"] Sep 11 15:24:37.125969 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:37.125943 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-34c71e-openshift-default-6b6887f464-sptnk"] Sep 11 15:24:38.099365 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:38.099334 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" event={"ID":"bb6925ff-2827-4653-bf35-be7f84df2a30","Type":"ContainerStarted","Data":"66ac0409a66a24f8f4f4895673cdd61b1192bbf87b25fda3edee31629adc20b9"} Sep 11 15:24:38.099856 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:38.099428 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:38.116616 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:38.116569 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" podStartSLOduration=1.703861346 podStartE2EDuration="2.116554241s" totalImagesPullingTime="412.692895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:24:38.115820796 +0000 UTC m=+972.347745228" watchObservedRunningTime="2026-09-11 15:24:38.116554241 +0000 UTC m=+972.348478672" Sep 11 15:24:38.264008 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:38.263971 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" path="/var/lib/kubelet/pods/eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02/volumes" Sep 11 15:24:40.587972 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:40.587941 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:44.110033 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:44.110000 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:24:44.185926 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:44.185890 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:24:46.968634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:46.968584 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6"] Sep 11 15:24:46.969265 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:46.969236 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9d80b0-cdc9-45bd-9d5b-7fc49248ce02" containerName="istio-proxy" Sep 11 15:24:46.973946 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:46.973919 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:46.977487 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:46.977455 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9f08bd\"/\"kube-root-ca.crt\"" Sep 11 15:24:46.977781 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:46.977733 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9f08bd\"/\"openshift-service-ca.crt\"" Sep 11 15:24:46.977982 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:46.977962 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9f08bd\"/\"default-dockercfg-twp9v\"" Sep 11 15:24:46.978349 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:46.978330 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9f08bd\"/\"test-model-9f08bd-kserve-self-signed-certs\"" Sep 11 15:24:46.982185 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:46.982155 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6"] Sep 11 15:24:47.096243 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.096203 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-tmp-dir\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.096442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.096265 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-dshm\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.096442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.096287 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-model-cache\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.096442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.096345 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-home\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.096442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.096398 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-tls-certs\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.096604 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.096467 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg4rr\" (UniqueName: \"kubernetes.io/projected/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-kube-api-access-kg4rr\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.176436 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.176374 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:47.197348 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.197315 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg4rr\" (UniqueName: \"kubernetes.io/projected/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-kube-api-access-kg4rr\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.197348 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.197356 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-tmp-dir\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.197594 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.197415 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-dshm\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.197594 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.197443 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-model-cache\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.197594 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.197475 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-home\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.197594 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.197516 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-tls-certs\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.197886 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.197861 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-model-cache\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.197886 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.197876 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-tmp-dir\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.197983 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.197932 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-home\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.199713 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.199691 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-dshm\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.200004 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.199981 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-tls-certs\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.207921 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.207899 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg4rr\" (UniqueName: \"kubernetes.io/projected/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-kube-api-access-kg4rr\") pod \"test-model-9f08bd-kserve-7f58c84f47-tgzb6\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.286756 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.286654 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:47.399311 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.399272 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:47.574424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:47.562624 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6"] Sep 11 15:24:47.574424 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:24:47.564384 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26fe2dd6_4ec4_4ae4_922d_06ed70204f35.slice/crio-83cdbf10289da556e78ddd22e0a05cff4b2e482789b7d92fc4ece6198c1a4524 WatchSource:0}: Error finding container 83cdbf10289da556e78ddd22e0a05cff4b2e482789b7d92fc4ece6198c1a4524: Status 404 returned error can't find the container with id 83cdbf10289da556e78ddd22e0a05cff4b2e482789b7d92fc4ece6198c1a4524 Sep 11 15:24:48.159215 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:48.159171 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" event={"ID":"26fe2dd6-4ec4-4ae4-922d-06ed70204f35","Type":"ContainerStarted","Data":"31b89d04050d86bce7c229507a0676c4c3749da1230bd50210a463f05b3eff90"} Sep 11 15:24:48.159667 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:48.159229 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" event={"ID":"26fe2dd6-4ec4-4ae4-922d-06ed70204f35","Type":"ContainerStarted","Data":"83cdbf10289da556e78ddd22e0a05cff4b2e482789b7d92fc4ece6198c1a4524"} Sep 11 15:24:48.178318 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:48.178262 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" podStartSLOduration=2.17824151 podStartE2EDuration="2.17824151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:24:48.175924338 +0000 UTC m=+982.407848769" watchObservedRunningTime="2026-09-11 15:24:48.17824151 +0000 UTC m=+982.410165944" Sep 11 15:24:48.603734 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:48.603634 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:48.705247 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:48.705147 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:53.209799 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:53.209763 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:53.327155 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:53.327119 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:53.412031 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:53.411987 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:55.206705 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:55.206677 2558 generic.go:356] "Generic (PLEG): container finished" podID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerID="dede56e3668d04ebcaa55679398510bf722ba556c1c2777ae8a9e3d9de0bf81f" exitCode=0 Sep 11 15:24:55.207139 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:55.206771 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" event={"ID":"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46","Type":"ContainerDied","Data":"dede56e3668d04ebcaa55679398510bf722ba556c1c2777ae8a9e3d9de0bf81f"} Sep 11 15:24:55.207182 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:55.207164 2558 scope.go:117] "RemoveContainer" containerID="dede56e3668d04ebcaa55679398510bf722ba556c1c2777ae8a9e3d9de0bf81f" Sep 11 15:24:55.208360 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:55.208326 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:24:55.208461 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:55.208397 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:24:55.208461 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:55.208430 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:24:56.214142 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:56.214106 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" event={"ID":"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46","Type":"ContainerStarted","Data":"3b29dd1b2cffc67503949d888bd3ccd4cc5b112c647c5a092bb8d47cf106de66"} Sep 11 15:24:56.908631 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:56.908589 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:24:56.908976 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:56.908952 2558 patch_prober.go:28] interesting pod/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 15:24:56.909073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:56.909042 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" podUID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 15:24:57.287698 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:57.287607 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:57.287698 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:57.287691 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:57.300267 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:57.300241 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:57.909284 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:57.909247 2558 patch_prober.go:28] interesting pod/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 15:24:57.909463 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:57.909316 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" podUID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 15:24:58.236486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:58.236402 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:24:58.513236 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:58.513143 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:58.908707 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:58.908677 2558 patch_prober.go:28] interesting pod/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 15:24:58.908886 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:58.908753 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" podUID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 15:24:59.207465 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:59.207379 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:59.907381 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:59.907347 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:24:59.909442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:59.909414 2558 patch_prober.go:28] interesting pod/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 15:24:59.909581 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:24:59.909473 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" podUID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 15:25:00.006826 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:00.006793 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:00.720850 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:00.720816 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:00.781710 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:00.781674 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:25:00.909131 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:00.909094 2558 patch_prober.go:28] interesting pod/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 15:25:00.909527 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:00.909155 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" podUID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 15:25:01.912852 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:01.912824 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:25:01.913237 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:01.913069 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:25:01.913871 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:01.913853 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:25:06.303845 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:06.303818 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:09.330521 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:09.330486 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:15.524938 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:15.524900 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:15.638258 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:15.638223 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:15.727336 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:15.727303 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:17.746353 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:17.746324 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:18.636780 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:18.636702 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:19.334025 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:19.333987 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:20.006909 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:20.006871 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:20.914974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:20.914945 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:21.481599 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:21.481565 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:25:26.280647 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:26.280616 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:27.042402 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:27.042368 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:28.352340 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.352306 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg"] Sep 11 15:25:28.352803 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.352544 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" podUID="bb6925ff-2827-4653-bf35-be7f84df2a30" containerName="maas-api" containerID="cri-o://66ac0409a66a24f8f4f4895673cdd61b1192bbf87b25fda3edee31629adc20b9" gracePeriod=30 Sep 11 15:25:28.382753 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.382715 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:28.451921 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.451890 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:28.551893 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.551850 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:28.608402 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.608349 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:25:28.631632 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.631600 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:28.775202 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.775163 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6925ff-2827-4653-bf35-be7f84df2a30-metrics-certs\") pod \"bb6925ff-2827-4653-bf35-be7f84df2a30\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " Sep 11 15:25:28.775376 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.775232 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4w94\" (UniqueName: \"kubernetes.io/projected/bb6925ff-2827-4653-bf35-be7f84df2a30-kube-api-access-m4w94\") pod \"bb6925ff-2827-4653-bf35-be7f84df2a30\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " Sep 11 15:25:28.775376 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.775262 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6925ff-2827-4653-bf35-be7f84df2a30-openshift-service-ca\") pod \"bb6925ff-2827-4653-bf35-be7f84df2a30\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " Sep 11 15:25:28.775376 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.775286 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb6925ff-2827-4653-bf35-be7f84df2a30-maas-api-tls\") pod \"bb6925ff-2827-4653-bf35-be7f84df2a30\" (UID: \"bb6925ff-2827-4653-bf35-be7f84df2a30\") " Sep 11 15:25:28.775658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.775628 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6925ff-2827-4653-bf35-be7f84df2a30-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb6925ff-2827-4653-bf35-be7f84df2a30" (UID: "bb6925ff-2827-4653-bf35-be7f84df2a30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:25:28.777580 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.777550 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb6925ff-2827-4653-bf35-be7f84df2a30-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb6925ff-2827-4653-bf35-be7f84df2a30" (UID: "bb6925ff-2827-4653-bf35-be7f84df2a30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:25:28.777998 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.777964 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6925ff-2827-4653-bf35-be7f84df2a30-kube-api-access-m4w94" (OuterVolumeSpecName: "kube-api-access-m4w94") pod "bb6925ff-2827-4653-bf35-be7f84df2a30" (UID: "bb6925ff-2827-4653-bf35-be7f84df2a30"). InnerVolumeSpecName "kube-api-access-m4w94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:25:28.777998 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.777985 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb6925ff-2827-4653-bf35-be7f84df2a30-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bb6925ff-2827-4653-bf35-be7f84df2a30" (UID: "bb6925ff-2827-4653-bf35-be7f84df2a30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:25:28.876553 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.876486 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb6925ff-2827-4653-bf35-be7f84df2a30-metrics-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:25:28.876553 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.876513 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4w94\" (UniqueName: \"kubernetes.io/projected/bb6925ff-2827-4653-bf35-be7f84df2a30-kube-api-access-m4w94\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:25:28.876553 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.876523 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6925ff-2827-4653-bf35-be7f84df2a30-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:25:28.876553 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.876532 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb6925ff-2827-4653-bf35-be7f84df2a30-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:25:28.916268 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:28.916244 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:29.017150 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:29.017079 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:29.390504 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:29.390468 2558 generic.go:356] "Generic (PLEG): container finished" podID="bb6925ff-2827-4653-bf35-be7f84df2a30" containerID="66ac0409a66a24f8f4f4895673cdd61b1192bbf87b25fda3edee31629adc20b9" exitCode=0 Sep 11 15:25:29.390956 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:29.390527 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" Sep 11 15:25:29.390956 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:29.390504 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" event={"ID":"bb6925ff-2827-4653-bf35-be7f84df2a30","Type":"ContainerDied","Data":"66ac0409a66a24f8f4f4895673cdd61b1192bbf87b25fda3edee31629adc20b9"} Sep 11 15:25:29.390956 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:29.390677 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg" event={"ID":"bb6925ff-2827-4653-bf35-be7f84df2a30","Type":"ContainerDied","Data":"7ca6d7dfd0ed1c528c2cfe0d5ba59f2b0b01778ea0ca703f3394f932a14a4678"} Sep 11 15:25:29.390956 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:29.390715 2558 scope.go:117] "RemoveContainer" containerID="66ac0409a66a24f8f4f4895673cdd61b1192bbf87b25fda3edee31629adc20b9" Sep 11 15:25:29.400984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:29.400967 2558 scope.go:117] "RemoveContainer" containerID="66ac0409a66a24f8f4f4895673cdd61b1192bbf87b25fda3edee31629adc20b9" Sep 11 15:25:29.401261 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:25:29.401227 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ac0409a66a24f8f4f4895673cdd61b1192bbf87b25fda3edee31629adc20b9\": container with ID starting with 66ac0409a66a24f8f4f4895673cdd61b1192bbf87b25fda3edee31629adc20b9 not found: ID does not exist" containerID="66ac0409a66a24f8f4f4895673cdd61b1192bbf87b25fda3edee31629adc20b9" Sep 11 15:25:29.401325 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:29.401273 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ac0409a66a24f8f4f4895673cdd61b1192bbf87b25fda3edee31629adc20b9"} err="failed to get container status \"66ac0409a66a24f8f4f4895673cdd61b1192bbf87b25fda3edee31629adc20b9\": rpc error: code = NotFound desc = could not find container \"66ac0409a66a24f8f4f4895673cdd61b1192bbf87b25fda3edee31629adc20b9\": container with ID starting with 66ac0409a66a24f8f4f4895673cdd61b1192bbf87b25fda3edee31629adc20b9 not found: ID does not exist" Sep 11 15:25:29.425898 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:29.425874 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg"] Sep 11 15:25:29.434640 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:29.434616 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ca78ef-6f74fb5f67-sxbrg"] Sep 11 15:25:29.832814 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:29.832778 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:30.264704 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:30.264672 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb6925ff-2827-4653-bf35-be7f84df2a30" path="/var/lib/kubelet/pods/bb6925ff-2827-4653-bf35-be7f84df2a30/volumes" Sep 11 15:25:30.501275 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:30.501241 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:32.694575 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:32.694542 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:32.750681 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:32.750647 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:32.827564 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:32.827534 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:32.924089 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:32.924048 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:33.023832 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:33.023759 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:33.684236 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:33.684206 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:33.776476 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:33.776446 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:33.876391 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:33.876353 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:33.977457 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:33.977376 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:35.463414 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:35.463387 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:36.683127 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:36.683098 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:25:39.756906 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:39.756870 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:39.984172 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:39.984141 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b"] Sep 11 15:25:39.984390 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:39.984349 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" podUID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerName="istio-proxy" containerID="cri-o://3b29dd1b2cffc67503949d888bd3ccd4cc5b112c647c5a092bb8d47cf106de66" gracePeriod=30 Sep 11 15:25:40.009250 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.009177 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:40.448477 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.448439 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:40.739795 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.739682 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89"] Sep 11 15:25:40.740283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.740267 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb6925ff-2827-4653-bf35-be7f84df2a30" containerName="maas-api" Sep 11 15:25:40.740330 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.740287 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb6925ff-2827-4653-bf35-be7f84df2a30" containerName="maas-api" Sep 11 15:25:40.740408 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.740391 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb6925ff-2827-4653-bf35-be7f84df2a30" containerName="maas-api" Sep 11 15:25:40.745383 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.745362 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.747624 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.747602 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-d3b1ea-openshift-default-dockercfg-6vgnt\"" Sep 11 15:25:40.753335 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.753046 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89"] Sep 11 15:25:40.760318 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.760296 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:40.886333 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.886299 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-workload-socket\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.886552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.886369 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/455dc369-2500-4791-8ea1-07ec4977b0fc-istiod-ca-cert\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.886552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.886435 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-envoy\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.886552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.886502 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-podinfo\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.886552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.886546 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-token\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.886786 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.886575 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42fbt\" (UniqueName: \"kubernetes.io/projected/455dc369-2500-4791-8ea1-07ec4977b0fc-kube-api-access-42fbt\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.886786 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.886677 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-workload-certs\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.886786 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.886720 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-data\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.886914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.886783 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-credential-socket\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.987517 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.987487 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-workload-certs\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.987517 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.987520 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-data\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.987793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.987545 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-credential-socket\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.987793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.987570 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-workload-socket\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.987793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.987641 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/455dc369-2500-4791-8ea1-07ec4977b0fc-istiod-ca-cert\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.987793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.987666 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-envoy\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.987793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.987697 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-podinfo\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.987793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.987731 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-token\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.987793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.987783 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42fbt\" (UniqueName: \"kubernetes.io/projected/455dc369-2500-4791-8ea1-07ec4977b0fc-kube-api-access-42fbt\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.988234 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.988030 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-workload-certs\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.988234 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.988062 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-credential-socket\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.988234 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.988086 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-data\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.988234 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.988109 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-workload-socket\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.988568 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.988544 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/455dc369-2500-4791-8ea1-07ec4977b0fc-istiod-ca-cert\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.990224 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.990172 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-envoy\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.990422 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.990403 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-podinfo\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.995536 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.995506 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-token\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:40.995659 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:40.995640 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42fbt\" (UniqueName: \"kubernetes.io/projected/455dc369-2500-4791-8ea1-07ec4977b0fc-kube-api-access-42fbt\") pod \"e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:41.057311 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:41.057286 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:41.317334 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:41.317231 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:41.335025 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:41.335004 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89"] Sep 11 15:25:41.336997 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:25:41.336969 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455dc369_2500_4791_8ea1_07ec4977b0fc.slice/crio-64fd37f8712464f502197df530de5705dd3640461bb6ef3c7bf02774dbdbeefa WatchSource:0}: Error finding container 64fd37f8712464f502197df530de5705dd3640461bb6ef3c7bf02774dbdbeefa: Status 404 returned error can't find the container with id 64fd37f8712464f502197df530de5705dd3640461bb6ef3c7bf02774dbdbeefa Sep 11 15:25:41.339186 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:41.339153 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:25:41.339278 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:41.339217 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:25:41.339278 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:41.339246 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:25:41.461687 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:41.461653 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" event={"ID":"455dc369-2500-4791-8ea1-07ec4977b0fc","Type":"ContainerStarted","Data":"85c1c5bf1b475360a92220c31d6657551eeea6a12f511916fa261c87028f67a6"} Sep 11 15:25:41.461868 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:41.461716 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" event={"ID":"455dc369-2500-4791-8ea1-07ec4977b0fc","Type":"ContainerStarted","Data":"64fd37f8712464f502197df530de5705dd3640461bb6ef3c7bf02774dbdbeefa"} Sep 11 15:25:41.483509 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:41.483449 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" podStartSLOduration=1.4834287800000001 podStartE2EDuration="1.48342878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:25:41.482044297 +0000 UTC m=+1035.713968729" watchObservedRunningTime="2026-09-11 15:25:41.48342878 +0000 UTC m=+1035.715353215" Sep 11 15:25:41.760362 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:41.760332 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:41.871151 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:41.871121 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:42.029052 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:42.028983 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:42.057940 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:42.057900 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:42.064500 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:42.064468 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:42.468301 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:42.468270 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:42.469216 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:42.469197 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:25:42.491829 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:42.491802 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:42.589586 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:42.589557 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:43.161935 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:43.161905 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:45.236896 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.236876 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:25:45.323987 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.323900 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-workload-socket\") pod \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " Sep 11 15:25:45.323987 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.323956 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-credential-socket\") pod \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " Sep 11 15:25:45.324203 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.323991 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-token\") pod \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " Sep 11 15:25:45.324203 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.324037 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-podinfo\") pod \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " Sep 11 15:25:45.324203 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.324068 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-data\") pod \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " Sep 11 15:25:45.324203 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.324105 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istiod-ca-cert\") pod \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " Sep 11 15:25:45.324203 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.324152 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgc8t\" (UniqueName: \"kubernetes.io/projected/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-kube-api-access-pgc8t\") pod \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " Sep 11 15:25:45.324203 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.324195 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-workload-certs\") pod \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " Sep 11 15:25:45.324518 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.324226 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-envoy\") pod \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\" (UID: \"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46\") " Sep 11 15:25:45.324518 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.324300 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" (UID: "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:25:45.324518 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.324311 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" (UID: "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:25:45.324687 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.324639 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:25:45.324687 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.324668 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:25:45.324838 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.324705 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" (UID: "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:25:45.324901 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.324840 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-data" (OuterVolumeSpecName: "istio-data") pod "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" (UID: "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:25:45.324963 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.324903 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" (UID: "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:25:45.327013 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.326977 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-kube-api-access-pgc8t" (OuterVolumeSpecName: "kube-api-access-pgc8t") pod "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" (UID: "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46"). InnerVolumeSpecName "kube-api-access-pgc8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:25:45.327013 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.326923 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" (UID: "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:25:45.327198 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.327015 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" (UID: "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:25:45.327247 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.327207 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" (UID: "ad748dcb-f098-4f4d-8bc9-bdbccf34bd46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:25:45.425233 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.425206 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:25:45.425233 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.425231 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:25:45.425233 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.425241 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:25:45.425449 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.425249 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:25:45.425449 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.425259 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgc8t\" (UniqueName: \"kubernetes.io/projected/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-kube-api-access-pgc8t\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:25:45.425449 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.425269 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:25:45.425449 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.425277 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:25:45.485974 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.485945 2558 generic.go:356] "Generic (PLEG): container finished" podID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerID="3b29dd1b2cffc67503949d888bd3ccd4cc5b112c647c5a092bb8d47cf106de66" exitCode=0 Sep 11 15:25:45.486146 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.486005 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" Sep 11 15:25:45.486146 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.486035 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" event={"ID":"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46","Type":"ContainerDied","Data":"3b29dd1b2cffc67503949d888bd3ccd4cc5b112c647c5a092bb8d47cf106de66"} Sep 11 15:25:45.486146 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.486088 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b" event={"ID":"ad748dcb-f098-4f4d-8bc9-bdbccf34bd46","Type":"ContainerDied","Data":"fed149992f95ac333b2a22d03f7533d5f8027d82efa31e653a9b7abe6c59dc6d"} Sep 11 15:25:45.486146 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.486111 2558 scope.go:117] "RemoveContainer" containerID="3b29dd1b2cffc67503949d888bd3ccd4cc5b112c647c5a092bb8d47cf106de66" Sep 11 15:25:45.500592 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.500560 2558 scope.go:117] "RemoveContainer" containerID="dede56e3668d04ebcaa55679398510bf722ba556c1c2777ae8a9e3d9de0bf81f" Sep 11 15:25:45.508797 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.508782 2558 scope.go:117] "RemoveContainer" containerID="3b29dd1b2cffc67503949d888bd3ccd4cc5b112c647c5a092bb8d47cf106de66" Sep 11 15:25:45.509097 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:25:45.509070 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b29dd1b2cffc67503949d888bd3ccd4cc5b112c647c5a092bb8d47cf106de66\": container with ID starting with 3b29dd1b2cffc67503949d888bd3ccd4cc5b112c647c5a092bb8d47cf106de66 not found: ID does not exist" containerID="3b29dd1b2cffc67503949d888bd3ccd4cc5b112c647c5a092bb8d47cf106de66" Sep 11 15:25:45.509191 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.509104 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b29dd1b2cffc67503949d888bd3ccd4cc5b112c647c5a092bb8d47cf106de66"} err="failed to get container status \"3b29dd1b2cffc67503949d888bd3ccd4cc5b112c647c5a092bb8d47cf106de66\": rpc error: code = NotFound desc = could not find container \"3b29dd1b2cffc67503949d888bd3ccd4cc5b112c647c5a092bb8d47cf106de66\": container with ID starting with 3b29dd1b2cffc67503949d888bd3ccd4cc5b112c647c5a092bb8d47cf106de66 not found: ID does not exist" Sep 11 15:25:45.509191 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.509124 2558 scope.go:117] "RemoveContainer" containerID="dede56e3668d04ebcaa55679398510bf722ba556c1c2777ae8a9e3d9de0bf81f" Sep 11 15:25:45.509651 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:25:45.509619 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dede56e3668d04ebcaa55679398510bf722ba556c1c2777ae8a9e3d9de0bf81f\": container with ID starting with dede56e3668d04ebcaa55679398510bf722ba556c1c2777ae8a9e3d9de0bf81f not found: ID does not exist" containerID="dede56e3668d04ebcaa55679398510bf722ba556c1c2777ae8a9e3d9de0bf81f" Sep 11 15:25:45.510018 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.509652 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dede56e3668d04ebcaa55679398510bf722ba556c1c2777ae8a9e3d9de0bf81f"} err="failed to get container status \"dede56e3668d04ebcaa55679398510bf722ba556c1c2777ae8a9e3d9de0bf81f\": rpc error: code = NotFound desc = could not find container \"dede56e3668d04ebcaa55679398510bf722ba556c1c2777ae8a9e3d9de0bf81f\": container with ID starting with dede56e3668d04ebcaa55679398510bf722ba556c1c2777ae8a9e3d9de0bf81f not found: ID does not exist" Sep 11 15:25:45.512260 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.512240 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b"] Sep 11 15:25:45.515470 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:45.515450 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-ca78ef-openshift-default-8f6cb97b-blp2b"] Sep 11 15:25:46.264557 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:46.264523 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" path="/var/lib/kubelet/pods/ad748dcb-f098-4f4d-8bc9-bdbccf34bd46/volumes" Sep 11 15:25:49.109117 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.109082 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:49.169445 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.169418 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:49.358083 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.358050 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:49.661687 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.661654 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5"] Sep 11 15:25:49.662038 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.662024 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerName="istio-proxy" Sep 11 15:25:49.662098 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.662040 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerName="istio-proxy" Sep 11 15:25:49.662141 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.662103 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerName="istio-proxy" Sep 11 15:25:49.662141 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.662112 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerName="istio-proxy" Sep 11 15:25:49.666598 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.666571 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.669823 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.669795 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0dd91403-openshift-default-dockercfg-ppx8m\"" Sep 11 15:25:49.682814 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.682791 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5"] Sep 11 15:25:49.704749 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.704687 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:49.754763 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.754708 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27v4\" (UniqueName: \"kubernetes.io/projected/b7ba39a4-259f-44eb-b57d-089649575906-kube-api-access-k27v4\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.754763 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.754755 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-workload-certs\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.754989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.754847 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ba39a4-259f-44eb-b57d-089649575906-istiod-ca-cert\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.754989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.754912 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-credential-socket\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.754989 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.754978 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ba39a4-259f-44eb-b57d-089649575906-istio-token\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.755157 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.755010 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-istio-envoy\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.755157 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.755085 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ba39a4-259f-44eb-b57d-089649575906-istio-podinfo\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.755157 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.755125 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-workload-socket\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.755157 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.755151 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-istio-data\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.855558 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.855530 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:49.855697 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.855627 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27v4\" (UniqueName: \"kubernetes.io/projected/b7ba39a4-259f-44eb-b57d-089649575906-kube-api-access-k27v4\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.855697 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.855657 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-workload-certs\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.855697 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.855681 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ba39a4-259f-44eb-b57d-089649575906-istiod-ca-cert\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.855874 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.855714 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-credential-socket\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.855874 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.855791 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ba39a4-259f-44eb-b57d-089649575906-istio-token\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.855874 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.855823 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-istio-envoy\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.856159 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.856137 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-credential-socket\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.856339 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.856154 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ba39a4-259f-44eb-b57d-089649575906-istio-podinfo\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.856462 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.856207 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-workload-certs\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.856462 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.856376 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-workload-socket\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.856462 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.856409 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-istio-data\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.856704 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.856651 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ba39a4-259f-44eb-b57d-089649575906-istiod-ca-cert\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.856704 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.856677 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-workload-socket\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.856820 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.856723 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-istio-data\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.858275 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.858255 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-istio-envoy\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.858411 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.858393 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ba39a4-259f-44eb-b57d-089649575906-istio-podinfo\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.866755 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.864006 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ba39a4-259f-44eb-b57d-089649575906-istio-token\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.866755 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.864068 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27v4\" (UniqueName: \"kubernetes.io/projected/b7ba39a4-259f-44eb-b57d-089649575906-kube-api-access-k27v4\") pod \"e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:49.978619 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:49.978554 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:50.253635 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:50.253611 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5"] Sep 11 15:25:50.255377 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:25:50.255347 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ba39a4_259f_44eb_b57d_089649575906.slice/crio-350bf2c63dae04e09fd58711999cab40de546242b8a248bf0d9785763bfc4024 WatchSource:0}: Error finding container 350bf2c63dae04e09fd58711999cab40de546242b8a248bf0d9785763bfc4024: Status 404 returned error can't find the container with id 350bf2c63dae04e09fd58711999cab40de546242b8a248bf0d9785763bfc4024 Sep 11 15:25:50.257519 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:50.257486 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:25:50.257608 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:50.257551 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:25:50.257608 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:50.257578 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:25:50.517857 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:50.517768 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" event={"ID":"b7ba39a4-259f-44eb-b57d-089649575906","Type":"ContainerStarted","Data":"bd9702a9bf0ef6de30d8f642a40a8aaba4f3b45554cf5243525304f893fc7bde"} Sep 11 15:25:50.517857 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:50.517824 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" event={"ID":"b7ba39a4-259f-44eb-b57d-089649575906","Type":"ContainerStarted","Data":"350bf2c63dae04e09fd58711999cab40de546242b8a248bf0d9785763bfc4024"} Sep 11 15:25:50.538139 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:50.538093 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" podStartSLOduration=1.538076354 podStartE2EDuration="1.538076354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:25:50.536250779 +0000 UTC m=+1044.768175211" watchObservedRunningTime="2026-09-11 15:25:50.538076354 +0000 UTC m=+1044.770000785" Sep 11 15:25:50.979637 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:50.979603 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:51.983993 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:51.983965 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:52.004692 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:52.004658 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:52.528797 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:52.528762 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:52.529844 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:52.529822 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:25:52.636075 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:52.636026 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:52.729516 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:52.729457 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:53.583011 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:53.582979 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:25:53.642413 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:53.642369 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:54.426642 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:54.426612 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:54.491825 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:54.491790 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:54.592001 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:54.591948 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:55.019419 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:55.019391 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:25:55.109893 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:25:55.109847 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:08.287101 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:08.287066 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:26:11.645975 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:11.645896 2558 generic.go:356] "Generic (PLEG): container finished" podID="b7ba39a4-259f-44eb-b57d-089649575906" containerID="bd9702a9bf0ef6de30d8f642a40a8aaba4f3b45554cf5243525304f893fc7bde" exitCode=0 Sep 11 15:26:11.645975 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:11.645934 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" event={"ID":"b7ba39a4-259f-44eb-b57d-089649575906","Type":"ContainerDied","Data":"bd9702a9bf0ef6de30d8f642a40a8aaba4f3b45554cf5243525304f893fc7bde"} Sep 11 15:26:11.646457 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:11.646281 2558 scope.go:117] "RemoveContainer" containerID="bd9702a9bf0ef6de30d8f642a40a8aaba4f3b45554cf5243525304f893fc7bde" Sep 11 15:26:11.647326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:11.647296 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:26:11.647399 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:11.647385 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:26:11.647453 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:11.647438 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:26:12.216313 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:12.216279 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:12.680551 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:12.680497 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" event={"ID":"b7ba39a4-259f-44eb-b57d-089649575906","Type":"ContainerStarted","Data":"51338a31c91ae6b1a3f484d86784d0e8f210a12972ee11d278435138f6ab7a77"} Sep 11 15:26:12.979374 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:12.979276 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:26:12.979558 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:12.979468 2558 patch_prober.go:28] interesting pod/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 15:26:12.979558 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:12.979527 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" podUID="b7ba39a4-259f-44eb-b57d-089649575906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 15:26:13.979341 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:13.979298 2558 patch_prober.go:28] interesting pod/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 15:26:13.979770 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:13.979369 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" podUID="b7ba39a4-259f-44eb-b57d-089649575906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 15:26:14.398851 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:14.398819 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:14.979545 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:14.979505 2558 patch_prober.go:28] interesting pod/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 15:26:14.979993 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:14.979580 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" podUID="b7ba39a4-259f-44eb-b57d-089649575906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 15:26:15.794763 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:15.794710 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:15.902190 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:15.902156 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89"] Sep 11 15:26:15.902442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:15.902416 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" podUID="455dc369-2500-4791-8ea1-07ec4977b0fc" containerName="istio-proxy" containerID="cri-o://85c1c5bf1b475360a92220c31d6657551eeea6a12f511916fa261c87028f67a6" gracePeriod=30 Sep 11 15:26:15.922508 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:15.922476 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:15.979762 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:15.979710 2558 patch_prober.go:28] interesting pod/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 15:26:15.980113 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:15.979793 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" podUID="b7ba39a4-259f-44eb-b57d-089649575906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 15:26:16.136333 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:16.136248 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:16.248893 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:16.248858 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:16.481194 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:16.481151 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:16.984135 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:16.984100 2558 patch_prober.go:28] interesting pod/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 15:26:16.984700 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:16.984672 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" podUID="b7ba39a4-259f-44eb-b57d-089649575906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 15:26:16.985205 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:16.984468 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:16.987196 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:16.987174 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6"] Sep 11 15:26:17.426110 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:17.426077 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:17.589563 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:17.589528 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:17.982818 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:17.982791 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:26:17.983046 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:17.983029 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:26:17.983693 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:17.983675 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:26:18.261381 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:18.261303 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:18.358544 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:18.358487 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:18.458781 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:18.458731 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:18.695543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:18.695509 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:19.059318 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:19.059214 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:19.811919 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:19.811885 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:20.284048 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:20.284014 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:26:20.483543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:20.483516 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:21.158852 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.158829 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:26:21.239253 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239179 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-credential-socket\") pod \"455dc369-2500-4791-8ea1-07ec4977b0fc\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " Sep 11 15:26:21.239253 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239247 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/455dc369-2500-4791-8ea1-07ec4977b0fc-istiod-ca-cert\") pod \"455dc369-2500-4791-8ea1-07ec4977b0fc\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " Sep 11 15:26:21.239424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239314 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-workload-socket\") pod \"455dc369-2500-4791-8ea1-07ec4977b0fc\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " Sep 11 15:26:21.239424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239349 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-envoy\") pod \"455dc369-2500-4791-8ea1-07ec4977b0fc\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " Sep 11 15:26:21.239424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239379 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-podinfo\") pod \"455dc369-2500-4791-8ea1-07ec4977b0fc\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " Sep 11 15:26:21.239424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239402 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42fbt\" (UniqueName: \"kubernetes.io/projected/455dc369-2500-4791-8ea1-07ec4977b0fc-kube-api-access-42fbt\") pod \"455dc369-2500-4791-8ea1-07ec4977b0fc\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " Sep 11 15:26:21.239626 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239434 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-token\") pod \"455dc369-2500-4791-8ea1-07ec4977b0fc\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " Sep 11 15:26:21.239626 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239463 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-data\") pod \"455dc369-2500-4791-8ea1-07ec4977b0fc\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " Sep 11 15:26:21.239626 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239473 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "455dc369-2500-4791-8ea1-07ec4977b0fc" (UID: "455dc369-2500-4791-8ea1-07ec4977b0fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:21.239626 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239493 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-workload-certs\") pod \"455dc369-2500-4791-8ea1-07ec4977b0fc\" (UID: \"455dc369-2500-4791-8ea1-07ec4977b0fc\") " Sep 11 15:26:21.239874 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239681 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "455dc369-2500-4791-8ea1-07ec4977b0fc" (UID: "455dc369-2500-4791-8ea1-07ec4977b0fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:21.239874 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239721 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455dc369-2500-4791-8ea1-07ec4977b0fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "455dc369-2500-4791-8ea1-07ec4977b0fc" (UID: "455dc369-2500-4791-8ea1-07ec4977b0fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:26:21.239994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239848 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "455dc369-2500-4791-8ea1-07ec4977b0fc" (UID: "455dc369-2500-4791-8ea1-07ec4977b0fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:21.239994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239877 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:21.239994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239948 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:21.239994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.239908 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "455dc369-2500-4791-8ea1-07ec4977b0fc" (UID: "455dc369-2500-4791-8ea1-07ec4977b0fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:21.241684 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.241656 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "455dc369-2500-4791-8ea1-07ec4977b0fc" (UID: "455dc369-2500-4791-8ea1-07ec4977b0fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:26:21.241816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.241714 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "455dc369-2500-4791-8ea1-07ec4977b0fc" (UID: "455dc369-2500-4791-8ea1-07ec4977b0fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:26:21.241816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.241731 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "455dc369-2500-4791-8ea1-07ec4977b0fc" (UID: "455dc369-2500-4791-8ea1-07ec4977b0fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:21.241816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.241798 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455dc369-2500-4791-8ea1-07ec4977b0fc-kube-api-access-42fbt" (OuterVolumeSpecName: "kube-api-access-42fbt") pod "455dc369-2500-4791-8ea1-07ec4977b0fc" (UID: "455dc369-2500-4791-8ea1-07ec4977b0fc"). InnerVolumeSpecName "kube-api-access-42fbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:26:21.265245 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.265221 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:21.340567 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.340541 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:21.340567 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.340561 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:21.340567 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.340571 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42fbt\" (UniqueName: \"kubernetes.io/projected/455dc369-2500-4791-8ea1-07ec4977b0fc-kube-api-access-42fbt\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:21.340772 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.340580 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:21.340772 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.340589 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:21.340772 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.340596 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/455dc369-2500-4791-8ea1-07ec4977b0fc-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:21.340772 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.340604 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/455dc369-2500-4791-8ea1-07ec4977b0fc-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:21.734161 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.734125 2558 generic.go:356] "Generic (PLEG): container finished" podID="455dc369-2500-4791-8ea1-07ec4977b0fc" containerID="85c1c5bf1b475360a92220c31d6657551eeea6a12f511916fa261c87028f67a6" exitCode=0 Sep 11 15:26:21.734347 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.734169 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" event={"ID":"455dc369-2500-4791-8ea1-07ec4977b0fc","Type":"ContainerDied","Data":"85c1c5bf1b475360a92220c31d6657551eeea6a12f511916fa261c87028f67a6"} Sep 11 15:26:21.734347 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.734174 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" Sep 11 15:26:21.734347 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.734209 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89" event={"ID":"455dc369-2500-4791-8ea1-07ec4977b0fc","Type":"ContainerDied","Data":"64fd37f8712464f502197df530de5705dd3640461bb6ef3c7bf02774dbdbeefa"} Sep 11 15:26:21.734347 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.734231 2558 scope.go:117] "RemoveContainer" containerID="85c1c5bf1b475360a92220c31d6657551eeea6a12f511916fa261c87028f67a6" Sep 11 15:26:21.744204 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.744185 2558 scope.go:117] "RemoveContainer" containerID="85c1c5bf1b475360a92220c31d6657551eeea6a12f511916fa261c87028f67a6" Sep 11 15:26:21.744469 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:26:21.744452 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c1c5bf1b475360a92220c31d6657551eeea6a12f511916fa261c87028f67a6\": container with ID starting with 85c1c5bf1b475360a92220c31d6657551eeea6a12f511916fa261c87028f67a6 not found: ID does not exist" containerID="85c1c5bf1b475360a92220c31d6657551eeea6a12f511916fa261c87028f67a6" Sep 11 15:26:21.744508 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.744477 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c1c5bf1b475360a92220c31d6657551eeea6a12f511916fa261c87028f67a6"} err="failed to get container status \"85c1c5bf1b475360a92220c31d6657551eeea6a12f511916fa261c87028f67a6\": rpc error: code = NotFound desc = could not find container \"85c1c5bf1b475360a92220c31d6657551eeea6a12f511916fa261c87028f67a6\": container with ID starting with 85c1c5bf1b475360a92220c31d6657551eeea6a12f511916fa261c87028f67a6 not found: ID does not exist" Sep 11 15:26:21.755999 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.755974 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89"] Sep 11 15:26:21.761782 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:21.761757 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-d3b1ea-openshift-default-6cbb45b7f6-t2x89"] Sep 11 15:26:22.263855 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:22.263825 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="455dc369-2500-4791-8ea1-07ec4977b0fc" path="/var/lib/kubelet/pods/455dc369-2500-4791-8ea1-07ec4977b0fc/volumes" Sep 11 15:26:24.672554 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:24.672514 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:24.765234 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:24.765199 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:25.229378 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:25.229351 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:26.112450 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:26.112413 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:26.781344 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:26.781307 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:27.686421 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:27.686383 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:28.374018 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:28.373987 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:28.959601 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:28.959564 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:29.059558 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:29.059528 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:29.152511 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:29.152468 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:29.208940 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:29.208902 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:29.525075 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:29.525040 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:29.625211 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:29.625178 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:30.322934 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:30.322899 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:31.280399 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:31.280366 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:32.006672 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.006610 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" podUID="26fe2dd6-4ec4-4ae4-922d-06ed70204f35" containerName="main" containerID="cri-o://31b89d04050d86bce7c229507a0676c4c3749da1230bd50210a463f05b3eff90" gracePeriod=45 Sep 11 15:26:32.250352 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.250328 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:26:32.282363 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.282291 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:26:32.440105 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.440076 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-dshm\") pod \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " Sep 11 15:26:32.440105 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.440109 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg4rr\" (UniqueName: \"kubernetes.io/projected/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-kube-api-access-kg4rr\") pod \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " Sep 11 15:26:32.440336 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.440149 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-tmp-dir\") pod \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " Sep 11 15:26:32.440336 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.440213 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-tls-certs\") pod \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " Sep 11 15:26:32.440463 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.440347 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-model-cache\") pod \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " Sep 11 15:26:32.440463 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.440430 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-home\") pod \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\" (UID: \"26fe2dd6-4ec4-4ae4-922d-06ed70204f35\") " Sep 11 15:26:32.440579 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.440431 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26fe2dd6-4ec4-4ae4-922d-06ed70204f35" (UID: "26fe2dd6-4ec4-4ae4-922d-06ed70204f35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:32.440579 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.440551 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-model-cache" (OuterVolumeSpecName: "model-cache") pod "26fe2dd6-4ec4-4ae4-922d-06ed70204f35" (UID: "26fe2dd6-4ec4-4ae4-922d-06ed70204f35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:32.440579 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.440565 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-home" (OuterVolumeSpecName: "home") pod "26fe2dd6-4ec4-4ae4-922d-06ed70204f35" (UID: "26fe2dd6-4ec4-4ae4-922d-06ed70204f35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:32.440861 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.440842 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-model-cache\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:32.440916 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.440867 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-home\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:32.440916 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.440881 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-tmp-dir\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:32.442606 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.442575 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-kube-api-access-kg4rr" (OuterVolumeSpecName: "kube-api-access-kg4rr") pod "26fe2dd6-4ec4-4ae4-922d-06ed70204f35" (UID: "26fe2dd6-4ec4-4ae4-922d-06ed70204f35"). InnerVolumeSpecName "kube-api-access-kg4rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:26:32.442606 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.442597 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-dshm" (OuterVolumeSpecName: "dshm") pod "26fe2dd6-4ec4-4ae4-922d-06ed70204f35" (UID: "26fe2dd6-4ec4-4ae4-922d-06ed70204f35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:32.442825 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.442631 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26fe2dd6-4ec4-4ae4-922d-06ed70204f35" (UID: "26fe2dd6-4ec4-4ae4-922d-06ed70204f35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:26:32.542395 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.542306 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-dshm\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:32.542395 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.542340 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg4rr\" (UniqueName: \"kubernetes.io/projected/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-kube-api-access-kg4rr\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:32.542395 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.542351 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26fe2dd6-4ec4-4ae4-922d-06ed70204f35-tls-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:32.798126 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.798036 2558 generic.go:356] "Generic (PLEG): container finished" podID="26fe2dd6-4ec4-4ae4-922d-06ed70204f35" containerID="31b89d04050d86bce7c229507a0676c4c3749da1230bd50210a463f05b3eff90" exitCode=0 Sep 11 15:26:32.798126 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.798078 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" Sep 11 15:26:32.798126 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.798112 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" event={"ID":"26fe2dd6-4ec4-4ae4-922d-06ed70204f35","Type":"ContainerDied","Data":"31b89d04050d86bce7c229507a0676c4c3749da1230bd50210a463f05b3eff90"} Sep 11 15:26:32.798375 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.798143 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6" event={"ID":"26fe2dd6-4ec4-4ae4-922d-06ed70204f35","Type":"ContainerDied","Data":"83cdbf10289da556e78ddd22e0a05cff4b2e482789b7d92fc4ece6198c1a4524"} Sep 11 15:26:32.798375 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.798160 2558 scope.go:117] "RemoveContainer" containerID="31b89d04050d86bce7c229507a0676c4c3749da1230bd50210a463f05b3eff90" Sep 11 15:26:32.807542 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.807526 2558 scope.go:117] "RemoveContainer" containerID="31b89d04050d86bce7c229507a0676c4c3749da1230bd50210a463f05b3eff90" Sep 11 15:26:32.807839 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:26:32.807821 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b89d04050d86bce7c229507a0676c4c3749da1230bd50210a463f05b3eff90\": container with ID starting with 31b89d04050d86bce7c229507a0676c4c3749da1230bd50210a463f05b3eff90 not found: ID does not exist" containerID="31b89d04050d86bce7c229507a0676c4c3749da1230bd50210a463f05b3eff90" Sep 11 15:26:32.807914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.807851 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b89d04050d86bce7c229507a0676c4c3749da1230bd50210a463f05b3eff90"} err="failed to get container status \"31b89d04050d86bce7c229507a0676c4c3749da1230bd50210a463f05b3eff90\": rpc error: code = NotFound desc = could not find container \"31b89d04050d86bce7c229507a0676c4c3749da1230bd50210a463f05b3eff90\": container with ID starting with 31b89d04050d86bce7c229507a0676c4c3749da1230bd50210a463f05b3eff90 not found: ID does not exist" Sep 11 15:26:32.819052 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.819031 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6"] Sep 11 15:26:32.821120 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:32.821100 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9f08bd/test-model-9f08bd-kserve-7f58c84f47-tgzb6"] Sep 11 15:26:34.264315 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:34.264285 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26fe2dd6-4ec4-4ae4-922d-06ed70204f35" path="/var/lib/kubelet/pods/26fe2dd6-4ec4-4ae4-922d-06ed70204f35/volumes" Sep 11 15:26:35.140454 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:35.140421 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:35.723214 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:35.723179 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:35.789758 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:35.789641 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:35.848970 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:35.848929 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:35.920387 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:35.920341 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:35.974578 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:35.974455 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:36.173842 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:36.173785 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:36.238282 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:36.238176 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:36.325991 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:36.325964 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:36.429094 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:36.429059 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:36.498591 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:36.498517 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:37.328565 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:37.328527 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:38.221862 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:38.221831 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:42.102757 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:42.102716 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:42.202791 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:42.202729 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:42.341300 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:42.341261 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5"] Sep 11 15:26:42.341557 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:42.341517 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" podUID="b7ba39a4-259f-44eb-b57d-089649575906" containerName="istio-proxy" containerID="cri-o://51338a31c91ae6b1a3f484d86784d0e8f210a12972ee11d278435138f6ab7a77" gracePeriod=30 Sep 11 15:26:42.363568 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:42.363486 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:42.758769 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:42.758709 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:42.839910 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:42.839876 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:42.970626 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:42.970593 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:44.179497 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:44.179463 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:44.585813 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:44.585703 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:26:46.969026 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:46.968995 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:47.517505 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.517463 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x"] Sep 11 15:26:47.517919 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.517698 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" podUID="1f410641-3c6f-40e7-b8f7-c4d05b623974" containerName="maas-api" containerID="cri-o://61efe59b224e2f93f63815a5be11e5a0bd2d73648a8339deb4e95c6c04b1a6b5" gracePeriod=30 Sep 11 15:26:47.551218 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.551136 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:47.631604 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.631575 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:26:47.652274 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.652239 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:47.675896 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.675869 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ba39a4-259f-44eb-b57d-089649575906-istio-token\") pod \"b7ba39a4-259f-44eb-b57d-089649575906\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " Sep 11 15:26:47.676068 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.675914 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-istio-data\") pod \"b7ba39a4-259f-44eb-b57d-089649575906\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " Sep 11 15:26:47.676068 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.675938 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k27v4\" (UniqueName: \"kubernetes.io/projected/b7ba39a4-259f-44eb-b57d-089649575906-kube-api-access-k27v4\") pod \"b7ba39a4-259f-44eb-b57d-089649575906\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " Sep 11 15:26:47.676068 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.675957 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ba39a4-259f-44eb-b57d-089649575906-istio-podinfo\") pod \"b7ba39a4-259f-44eb-b57d-089649575906\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " Sep 11 15:26:47.676068 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.675985 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-istio-envoy\") pod \"b7ba39a4-259f-44eb-b57d-089649575906\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " Sep 11 15:26:47.676303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.676072 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-workload-certs\") pod \"b7ba39a4-259f-44eb-b57d-089649575906\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " Sep 11 15:26:47.676303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.676110 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-workload-socket\") pod \"b7ba39a4-259f-44eb-b57d-089649575906\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " Sep 11 15:26:47.676303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.676155 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-istio-data" (OuterVolumeSpecName: "istio-data") pod "b7ba39a4-259f-44eb-b57d-089649575906" (UID: "b7ba39a4-259f-44eb-b57d-089649575906"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:47.676303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.676166 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ba39a4-259f-44eb-b57d-089649575906-istiod-ca-cert\") pod \"b7ba39a4-259f-44eb-b57d-089649575906\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " Sep 11 15:26:47.676303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.676218 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-credential-socket\") pod \"b7ba39a4-259f-44eb-b57d-089649575906\" (UID: \"b7ba39a4-259f-44eb-b57d-089649575906\") " Sep 11 15:26:47.676596 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.676503 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b7ba39a4-259f-44eb-b57d-089649575906" (UID: "b7ba39a4-259f-44eb-b57d-089649575906"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:47.676596 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.676514 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ba39a4-259f-44eb-b57d-089649575906-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b7ba39a4-259f-44eb-b57d-089649575906" (UID: "b7ba39a4-259f-44eb-b57d-089649575906"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:26:47.676596 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.676538 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:47.676596 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.676566 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b7ba39a4-259f-44eb-b57d-089649575906" (UID: "b7ba39a4-259f-44eb-b57d-089649575906"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:47.676920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.676724 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b7ba39a4-259f-44eb-b57d-089649575906" (UID: "b7ba39a4-259f-44eb-b57d-089649575906"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:47.678461 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.678432 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ba39a4-259f-44eb-b57d-089649575906-kube-api-access-k27v4" (OuterVolumeSpecName: "kube-api-access-k27v4") pod "b7ba39a4-259f-44eb-b57d-089649575906" (UID: "b7ba39a4-259f-44eb-b57d-089649575906"). InnerVolumeSpecName "kube-api-access-k27v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:26:47.678461 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.678442 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ba39a4-259f-44eb-b57d-089649575906-istio-token" (OuterVolumeSpecName: "istio-token") pod "b7ba39a4-259f-44eb-b57d-089649575906" (UID: "b7ba39a4-259f-44eb-b57d-089649575906"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:26:47.678634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.678617 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7ba39a4-259f-44eb-b57d-089649575906-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b7ba39a4-259f-44eb-b57d-089649575906" (UID: "b7ba39a4-259f-44eb-b57d-089649575906"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:26:47.678781 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.678763 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b7ba39a4-259f-44eb-b57d-089649575906" (UID: "b7ba39a4-259f-44eb-b57d-089649575906"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:26:47.716370 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.716337 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:47.775850 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.775822 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:47.776994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.776971 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:47.777100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.776997 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:47.777100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.777011 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ba39a4-259f-44eb-b57d-089649575906-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:47.777100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.777023 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:47.777100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.777035 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ba39a4-259f-44eb-b57d-089649575906-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:47.777100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.777048 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k27v4\" (UniqueName: \"kubernetes.io/projected/b7ba39a4-259f-44eb-b57d-089649575906-kube-api-access-k27v4\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:47.777100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.777060 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ba39a4-259f-44eb-b57d-089649575906-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:47.777100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.777072 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ba39a4-259f-44eb-b57d-089649575906-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:47.885958 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.885874 2558 generic.go:356] "Generic (PLEG): container finished" podID="1f410641-3c6f-40e7-b8f7-c4d05b623974" containerID="61efe59b224e2f93f63815a5be11e5a0bd2d73648a8339deb4e95c6c04b1a6b5" exitCode=0 Sep 11 15:26:47.885958 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.885947 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" event={"ID":"1f410641-3c6f-40e7-b8f7-c4d05b623974","Type":"ContainerDied","Data":"61efe59b224e2f93f63815a5be11e5a0bd2d73648a8339deb4e95c6c04b1a6b5"} Sep 11 15:26:47.889032 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.889004 2558 generic.go:356] "Generic (PLEG): container finished" podID="b7ba39a4-259f-44eb-b57d-089649575906" containerID="51338a31c91ae6b1a3f484d86784d0e8f210a12972ee11d278435138f6ab7a77" exitCode=0 Sep 11 15:26:47.889180 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.889088 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" Sep 11 15:26:47.889180 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.889115 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" event={"ID":"b7ba39a4-259f-44eb-b57d-089649575906","Type":"ContainerDied","Data":"51338a31c91ae6b1a3f484d86784d0e8f210a12972ee11d278435138f6ab7a77"} Sep 11 15:26:47.889180 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.889161 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5" event={"ID":"b7ba39a4-259f-44eb-b57d-089649575906","Type":"ContainerDied","Data":"350bf2c63dae04e09fd58711999cab40de546242b8a248bf0d9785763bfc4024"} Sep 11 15:26:47.889332 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.889189 2558 scope.go:117] "RemoveContainer" containerID="51338a31c91ae6b1a3f484d86784d0e8f210a12972ee11d278435138f6ab7a77" Sep 11 15:26:47.904670 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.904650 2558 scope.go:117] "RemoveContainer" containerID="bd9702a9bf0ef6de30d8f642a40a8aaba4f3b45554cf5243525304f893fc7bde" Sep 11 15:26:47.914175 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.914150 2558 scope.go:117] "RemoveContainer" containerID="51338a31c91ae6b1a3f484d86784d0e8f210a12972ee11d278435138f6ab7a77" Sep 11 15:26:47.914576 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:26:47.914546 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51338a31c91ae6b1a3f484d86784d0e8f210a12972ee11d278435138f6ab7a77\": container with ID starting with 51338a31c91ae6b1a3f484d86784d0e8f210a12972ee11d278435138f6ab7a77 not found: ID does not exist" containerID="51338a31c91ae6b1a3f484d86784d0e8f210a12972ee11d278435138f6ab7a77" Sep 11 15:26:47.914674 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.914588 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51338a31c91ae6b1a3f484d86784d0e8f210a12972ee11d278435138f6ab7a77"} err="failed to get container status \"51338a31c91ae6b1a3f484d86784d0e8f210a12972ee11d278435138f6ab7a77\": rpc error: code = NotFound desc = could not find container \"51338a31c91ae6b1a3f484d86784d0e8f210a12972ee11d278435138f6ab7a77\": container with ID starting with 51338a31c91ae6b1a3f484d86784d0e8f210a12972ee11d278435138f6ab7a77 not found: ID does not exist" Sep 11 15:26:47.914674 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.914614 2558 scope.go:117] "RemoveContainer" containerID="bd9702a9bf0ef6de30d8f642a40a8aaba4f3b45554cf5243525304f893fc7bde" Sep 11 15:26:47.914992 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:26:47.914968 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9702a9bf0ef6de30d8f642a40a8aaba4f3b45554cf5243525304f893fc7bde\": container with ID starting with bd9702a9bf0ef6de30d8f642a40a8aaba4f3b45554cf5243525304f893fc7bde not found: ID does not exist" containerID="bd9702a9bf0ef6de30d8f642a40a8aaba4f3b45554cf5243525304f893fc7bde" Sep 11 15:26:47.915074 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.915002 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9702a9bf0ef6de30d8f642a40a8aaba4f3b45554cf5243525304f893fc7bde"} err="failed to get container status \"bd9702a9bf0ef6de30d8f642a40a8aaba4f3b45554cf5243525304f893fc7bde\": rpc error: code = NotFound desc = could not find container \"bd9702a9bf0ef6de30d8f642a40a8aaba4f3b45554cf5243525304f893fc7bde\": container with ID starting with bd9702a9bf0ef6de30d8f642a40a8aaba4f3b45554cf5243525304f893fc7bde not found: ID does not exist" Sep 11 15:26:47.918648 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.918625 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5"] Sep 11 15:26:47.923437 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.923413 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0dd91403-openshift-default-67cd79ddf6-5wll5"] Sep 11 15:26:47.957593 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:47.957571 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:26:48.079391 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.079362 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-maas-api-tls\") pod \"1f410641-3c6f-40e7-b8f7-c4d05b623974\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " Sep 11 15:26:48.079834 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.079418 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7m6j\" (UniqueName: \"kubernetes.io/projected/1f410641-3c6f-40e7-b8f7-c4d05b623974-kube-api-access-c7m6j\") pod \"1f410641-3c6f-40e7-b8f7-c4d05b623974\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " Sep 11 15:26:48.079834 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.079542 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-metrics-certs\") pod \"1f410641-3c6f-40e7-b8f7-c4d05b623974\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " Sep 11 15:26:48.079834 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.079605 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f410641-3c6f-40e7-b8f7-c4d05b623974-openshift-service-ca\") pod \"1f410641-3c6f-40e7-b8f7-c4d05b623974\" (UID: \"1f410641-3c6f-40e7-b8f7-c4d05b623974\") " Sep 11 15:26:48.080053 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.080025 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f410641-3c6f-40e7-b8f7-c4d05b623974-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f410641-3c6f-40e7-b8f7-c4d05b623974" (UID: "1f410641-3c6f-40e7-b8f7-c4d05b623974"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:26:48.081876 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.081847 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f410641-3c6f-40e7-b8f7-c4d05b623974-kube-api-access-c7m6j" (OuterVolumeSpecName: "kube-api-access-c7m6j") pod "1f410641-3c6f-40e7-b8f7-c4d05b623974" (UID: "1f410641-3c6f-40e7-b8f7-c4d05b623974"). InnerVolumeSpecName "kube-api-access-c7m6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:26:48.082074 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.082050 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f410641-3c6f-40e7-b8f7-c4d05b623974" (UID: "1f410641-3c6f-40e7-b8f7-c4d05b623974"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:26:48.082074 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.082067 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f410641-3c6f-40e7-b8f7-c4d05b623974" (UID: "1f410641-3c6f-40e7-b8f7-c4d05b623974"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:26:48.124957 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.124924 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:48.180962 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.180930 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f410641-3c6f-40e7-b8f7-c4d05b623974-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:48.180962 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.180956 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:48.180962 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.180965 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7m6j\" (UniqueName: \"kubernetes.io/projected/1f410641-3c6f-40e7-b8f7-c4d05b623974-kube-api-access-c7m6j\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:48.181186 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.180974 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f410641-3c6f-40e7-b8f7-c4d05b623974-metrics-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:26:48.224584 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.224553 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:48.263409 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.263377 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7ba39a4-259f-44eb-b57d-089649575906" path="/var/lib/kubelet/pods/b7ba39a4-259f-44eb-b57d-089649575906/volumes" Sep 11 15:26:48.895875 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.895846 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" Sep 11 15:26:48.896065 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.895850 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x" event={"ID":"1f410641-3c6f-40e7-b8f7-c4d05b623974","Type":"ContainerDied","Data":"c64ed527d495d030bcea7c41b368816aa7a627c1ff1149035625c56343c1018d"} Sep 11 15:26:48.896065 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.895976 2558 scope.go:117] "RemoveContainer" containerID="61efe59b224e2f93f63815a5be11e5a0bd2d73648a8339deb4e95c6c04b1a6b5" Sep 11 15:26:48.913646 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.913621 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x"] Sep 11 15:26:48.919489 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:48.919453 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9f08bd-686b6d78d4-ngk5x"] Sep 11 15:26:49.115785 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:49.115726 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:49.177869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:49.177723 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:49.255890 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:49.255856 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:49.354293 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:49.354233 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:49.538401 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:49.538308 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:50.265116 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:50.265077 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f410641-3c6f-40e7-b8f7-c4d05b623974" path="/var/lib/kubelet/pods/1f410641-3c6f-40e7-b8f7-c4d05b623974/volumes" Sep 11 15:26:50.887973 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:50.887923 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:26:53.979692 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:53.979658 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:54.040853 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:54.040817 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:54.249852 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:54.249766 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:54.811271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:54.811242 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:55.095347 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095263 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc"] Sep 11 15:26:55.095687 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095647 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f410641-3c6f-40e7-b8f7-c4d05b623974" containerName="maas-api" Sep 11 15:26:55.095687 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095659 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f410641-3c6f-40e7-b8f7-c4d05b623974" containerName="maas-api" Sep 11 15:26:55.095687 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095668 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7ba39a4-259f-44eb-b57d-089649575906" containerName="istio-proxy" Sep 11 15:26:55.095687 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095674 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7ba39a4-259f-44eb-b57d-089649575906" containerName="istio-proxy" Sep 11 15:26:55.095932 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095691 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26fe2dd6-4ec4-4ae4-922d-06ed70204f35" containerName="main" Sep 11 15:26:55.095932 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095698 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="26fe2dd6-4ec4-4ae4-922d-06ed70204f35" containerName="main" Sep 11 15:26:55.095932 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095703 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerName="istio-proxy" Sep 11 15:26:55.095932 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095708 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad748dcb-f098-4f4d-8bc9-bdbccf34bd46" containerName="istio-proxy" Sep 11 15:26:55.095932 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095717 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="455dc369-2500-4791-8ea1-07ec4977b0fc" containerName="istio-proxy" Sep 11 15:26:55.095932 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095723 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="455dc369-2500-4791-8ea1-07ec4977b0fc" containerName="istio-proxy" Sep 11 15:26:55.095932 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095792 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7ba39a4-259f-44eb-b57d-089649575906" containerName="istio-proxy" Sep 11 15:26:55.095932 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095800 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="455dc369-2500-4791-8ea1-07ec4977b0fc" containerName="istio-proxy" Sep 11 15:26:55.095932 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095808 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f410641-3c6f-40e7-b8f7-c4d05b623974" containerName="maas-api" Sep 11 15:26:55.095932 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.095815 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="26fe2dd6-4ec4-4ae4-922d-06ed70204f35" containerName="main" Sep 11 15:26:55.098721 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.098698 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.101325 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.101301 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-397e40-openshift-default-dockercfg-8r4fg\"" Sep 11 15:26:55.111231 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.111205 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc"] Sep 11 15:26:55.120465 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.120437 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:55.145386 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.145346 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-token\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.145386 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.145410 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-credential-socket\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.145627 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.145454 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h6gq\" (UniqueName: \"kubernetes.io/projected/15ed46cf-24b8-450d-b140-21754d7bc97e-kube-api-access-8h6gq\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.145627 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.145507 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-workload-socket\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.145627 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.145545 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ed46cf-24b8-450d-b140-21754d7bc97e-istiod-ca-cert\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.145627 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.145569 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-podinfo\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.145627 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.145614 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-workload-certs\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.145821 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.145638 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-envoy\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.145821 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.145663 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-data\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.247030 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.246991 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ed46cf-24b8-450d-b140-21754d7bc97e-istiod-ca-cert\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.247030 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.247031 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-podinfo\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.247298 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.247066 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-workload-certs\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.247298 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.247088 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-envoy\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.247298 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.247111 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-data\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.247298 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.247142 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-token\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.247298 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.247194 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-credential-socket\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.247298 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.247219 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h6gq\" (UniqueName: \"kubernetes.io/projected/15ed46cf-24b8-450d-b140-21754d7bc97e-kube-api-access-8h6gq\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.247298 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.247250 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-workload-socket\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.248195 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.248153 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-workload-certs\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.248316 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.248211 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-data\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.248316 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.248230 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-workload-socket\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.248433 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.248401 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-credential-socket\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.248689 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.248667 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ed46cf-24b8-450d-b140-21754d7bc97e-istiod-ca-cert\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.250086 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.250058 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-envoy\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.250340 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.250325 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-podinfo\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.255670 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.255642 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-token\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.255847 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.255829 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h6gq\" (UniqueName: \"kubernetes.io/projected/15ed46cf-24b8-450d-b140-21754d7bc97e-kube-api-access-8h6gq\") pod \"e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.369699 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.369615 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:55.412149 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.412113 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:55.436055 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.436019 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:55.506019 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.505982 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:55.690913 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.690884 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc"] Sep 11 15:26:55.693411 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:26:55.693382 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ed46cf_24b8_450d_b140_21754d7bc97e.slice/crio-4d29189dc2168de8cdfc867cfce4e62208083580738646d5a72d100950eb4cd8 WatchSource:0}: Error finding container 4d29189dc2168de8cdfc867cfce4e62208083580738646d5a72d100950eb4cd8: Status 404 returned error can't find the container with id 4d29189dc2168de8cdfc867cfce4e62208083580738646d5a72d100950eb4cd8 Sep 11 15:26:55.695580 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.695542 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:26:55.695667 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.695608 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:26:55.695667 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.695651 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:26:55.941493 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.941395 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" event={"ID":"15ed46cf-24b8-450d-b140-21754d7bc97e","Type":"ContainerStarted","Data":"607dfcd28105639d6be82fa4e6b71266f755aa41b6d6e748b17187b19ed3ebbc"} Sep 11 15:26:55.941493 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.941447 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" event={"ID":"15ed46cf-24b8-450d-b140-21754d7bc97e","Type":"ContainerStarted","Data":"4d29189dc2168de8cdfc867cfce4e62208083580738646d5a72d100950eb4cd8"} Sep 11 15:26:55.960279 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:55.960225 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" podStartSLOduration=0.960204891 podStartE2EDuration="960.204891ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:26:55.958248218 +0000 UTC m=+1110.190172649" watchObservedRunningTime="2026-09-11 15:26:55.960204891 +0000 UTC m=+1110.192129323" Sep 11 15:26:56.269867 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.269770 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:56.412574 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.412532 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:56.417860 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.417832 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:56.554674 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.554588 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2"] Sep 11 15:26:56.555011 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.554992 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7ba39a4-259f-44eb-b57d-089649575906" containerName="istio-proxy" Sep 11 15:26:56.555075 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.555014 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7ba39a4-259f-44eb-b57d-089649575906" containerName="istio-proxy" Sep 11 15:26:56.555130 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.555111 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7ba39a4-259f-44eb-b57d-089649575906" containerName="istio-proxy" Sep 11 15:26:56.558230 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.558209 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.560466 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.560440 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-4a348b-openshift-default-dockercfg-6g7ch\"" Sep 11 15:26:56.576818 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.576532 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2"] Sep 11 15:26:56.592408 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.592372 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:56.659949 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.659914 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-credential-socket\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.659949 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.659952 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-workload-certs\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.660154 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.659992 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-data\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.660154 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.660096 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-workload-socket\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.660221 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.660167 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkgmr\" (UniqueName: \"kubernetes.io/projected/b78f8075-db10-4642-8d1f-cd7850ede4ef-kube-api-access-tkgmr\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.660221 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.660198 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-envoy\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.660295 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.660281 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-podinfo\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.660333 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.660302 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-token\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.660333 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.660324 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78f8075-db10-4642-8d1f-cd7850ede4ef-istiod-ca-cert\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.691543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.691506 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:56.761396 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.761361 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-podinfo\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.761617 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.761422 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-token\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.761617 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.761451 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78f8075-db10-4642-8d1f-cd7850ede4ef-istiod-ca-cert\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.761617 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.761494 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-credential-socket\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.761617 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.761520 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-workload-certs\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.761617 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.761579 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-data\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.761617 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.761609 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-workload-socket\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.761944 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.761639 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkgmr\" (UniqueName: \"kubernetes.io/projected/b78f8075-db10-4642-8d1f-cd7850ede4ef-kube-api-access-tkgmr\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.762006 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.761990 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-envoy\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.762064 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.762018 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-credential-socket\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.762454 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.762206 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-data\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.762454 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.762213 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-workload-socket\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.762454 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.762316 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78f8075-db10-4642-8d1f-cd7850ede4ef-istiod-ca-cert\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.762454 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.762393 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-workload-certs\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.764148 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.764122 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-envoy\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.764241 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.764163 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-podinfo\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.769393 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.769364 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-token\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.769608 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.769591 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkgmr\" (UniqueName: \"kubernetes.io/projected/b78f8075-db10-4642-8d1f-cd7850ede4ef-kube-api-access-tkgmr\") pod \"e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.855583 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.855503 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:56.871861 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.871821 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:56.948829 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.948802 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:56.949860 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:56.949838 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:26:57.153573 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:57.153544 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2"] Sep 11 15:26:57.154965 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:26:57.154939 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78f8075_db10_4642_8d1f_cd7850ede4ef.slice/crio-dc3d2e92d8be1a6476ca481cfe920b8bf01ed4333f77c9659a45047238f0a5bc WatchSource:0}: Error finding container dc3d2e92d8be1a6476ca481cfe920b8bf01ed4333f77c9659a45047238f0a5bc: Status 404 returned error can't find the container with id dc3d2e92d8be1a6476ca481cfe920b8bf01ed4333f77c9659a45047238f0a5bc Sep 11 15:26:57.157119 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:57.157086 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:26:57.157187 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:57.157151 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:26:57.157228 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:57.157185 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:26:57.956795 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:57.956713 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" event={"ID":"b78f8075-db10-4642-8d1f-cd7850ede4ef","Type":"ContainerStarted","Data":"b9ae2bc09f49ce20444f297287697c99bdbab9fa84df59044f96e1220f8b6532"} Sep 11 15:26:57.957272 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:57.956813 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" event={"ID":"b78f8075-db10-4642-8d1f-cd7850ede4ef","Type":"ContainerStarted","Data":"dc3d2e92d8be1a6476ca481cfe920b8bf01ed4333f77c9659a45047238f0a5bc"} Sep 11 15:26:57.986761 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:57.986694 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" podStartSLOduration=1.986677415 podStartE2EDuration="1.986677415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:26:57.982910323 +0000 UTC m=+1112.214834766" watchObservedRunningTime="2026-09-11 15:26:57.986677415 +0000 UTC m=+1112.218601846" Sep 11 15:26:57.998682 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:57.998644 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:58.078051 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:58.078017 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:58.522916 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:58.522876 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:58.872206 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:58.872114 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:58.877515 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:58.877487 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:58.962584 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:58.962549 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:58.963618 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:58.963596 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:26:59.242676 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:59.242642 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:59.344158 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:59.344122 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:59.803166 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:59.803130 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:59.903271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:59.903230 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:26:59.991342 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:26:59.991311 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:00.319167 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:00.319116 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:00.421005 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:00.420970 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:01.042816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:01.042785 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:01.832223 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:01.832195 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:01.934103 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:01.934073 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:02.033866 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:02.033832 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:02.306583 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:02.306552 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:02.390365 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:02.390334 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:03.817883 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:03.817848 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:27:04.848154 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:04.848108 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:12.990957 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:12.990907 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 15:27:12.991392 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:12.990993 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 15:27:13.383884 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:13.383802 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:27:14.658329 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:14.658146 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:14.716158 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:14.716111 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:15.059937 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:15.059895 2558 generic.go:356] "Generic (PLEG): container finished" podID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerID="607dfcd28105639d6be82fa4e6b71266f755aa41b6d6e748b17187b19ed3ebbc" exitCode=0 Sep 11 15:27:15.060191 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:15.060016 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" event={"ID":"15ed46cf-24b8-450d-b140-21754d7bc97e","Type":"ContainerDied","Data":"607dfcd28105639d6be82fa4e6b71266f755aa41b6d6e748b17187b19ed3ebbc"} Sep 11 15:27:15.060508 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:15.060482 2558 scope.go:117] "RemoveContainer" containerID="607dfcd28105639d6be82fa4e6b71266f755aa41b6d6e748b17187b19ed3ebbc" Sep 11 15:27:15.062066 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:15.061968 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:27:15.062363 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:15.062220 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:27:15.062551 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:15.062481 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:27:16.070917 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:16.070870 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" event={"ID":"15ed46cf-24b8-450d-b140-21754d7bc97e","Type":"ContainerStarted","Data":"6eac7eaf51c5197eea045bec46542a3c95f603a2478d77fc38daec1f833ee994"} Sep 11 15:27:16.412818 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:16.412773 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:27:16.413021 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:16.412981 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 15:27:16.413080 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:16.413028 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 15:27:17.413524 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:17.413479 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 15:27:17.414025 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:17.413545 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 15:27:18.413153 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:18.413120 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 15:27:18.413325 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:18.413174 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 15:27:19.412701 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:19.412663 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 15:27:19.413099 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:19.412722 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 15:27:20.379595 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:20.379548 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:20.412884 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:20.412850 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 15:27:20.415105 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:20.412925 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 15:27:21.104041 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:21.104009 2558 generic.go:356] "Generic (PLEG): container finished" podID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerID="b9ae2bc09f49ce20444f297287697c99bdbab9fa84df59044f96e1220f8b6532" exitCode=0 Sep 11 15:27:21.104212 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:21.104064 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" event={"ID":"b78f8075-db10-4642-8d1f-cd7850ede4ef","Type":"ContainerDied","Data":"b9ae2bc09f49ce20444f297287697c99bdbab9fa84df59044f96e1220f8b6532"} Sep 11 15:27:21.104441 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:21.104429 2558 scope.go:117] "RemoveContainer" containerID="b9ae2bc09f49ce20444f297287697c99bdbab9fa84df59044f96e1220f8b6532" Sep 11 15:27:21.105511 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:21.105480 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:27:21.105579 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:21.105562 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:27:21.105616 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:21.105603 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:27:21.412787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:21.412721 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 15:27:21.412953 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:21.412820 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 15:27:21.701357 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:21.701259 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:22.112184 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:22.112096 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" event={"ID":"b78f8075-db10-4642-8d1f-cd7850ede4ef","Type":"ContainerStarted","Data":"702ff745be88bb54f918f397dda079ea5f8c05a896f5359333dcbf1975777179"} Sep 11 15:27:22.416399 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:22.416370 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:27:22.416782 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:22.416675 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:27:22.417435 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:22.417414 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:27:22.532508 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:22.532478 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:22.872824 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:22.872704 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:27:22.873176 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:22.873080 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 15:27:22.873343 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:22.873318 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" podUID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 15:27:22.927139 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:22.927107 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:23.258845 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.258803 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:23.360670 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.360628 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:23.561825 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.561323 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594"] Sep 11 15:27:23.566056 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.566031 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.569060 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.569035 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-abba31-openshift-default-dockercfg-pckft\"" Sep 11 15:27:23.597760 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.594278 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594"] Sep 11 15:27:23.597760 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.595689 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:23.631091 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.631060 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-data\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.631091 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.631094 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-workload-socket\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.631283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.631120 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-token\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.631283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.631167 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-workload-certs\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.631283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.631196 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-podinfo\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.631283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.631234 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdh8w\" (UniqueName: \"kubernetes.io/projected/cecda5b5-849e-4c16-9258-9a2fd54555ac-kube-api-access-bdh8w\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.631283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.631257 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-credential-socket\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.631283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.631279 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecda5b5-849e-4c16-9258-9a2fd54555ac-istiod-ca-cert\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.631479 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.631414 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-envoy\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.732862 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.732826 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-envoy\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.733060 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.732870 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-data\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.733060 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.732900 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-workload-socket\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.733060 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.732928 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-token\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.733060 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.732951 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-workload-certs\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.733060 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.732972 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-podinfo\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.733348 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.733199 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdh8w\" (UniqueName: \"kubernetes.io/projected/cecda5b5-849e-4c16-9258-9a2fd54555ac-kube-api-access-bdh8w\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.733348 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.733309 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-workload-socket\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.733467 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.733389 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-workload-certs\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.733589 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.733564 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-credential-socket\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.733718 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.733591 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecda5b5-849e-4c16-9258-9a2fd54555ac-istiod-ca-cert\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.733718 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.733598 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-data\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.733881 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.733859 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-credential-socket\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.734147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.734128 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecda5b5-849e-4c16-9258-9a2fd54555ac-istiod-ca-cert\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.735221 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.735198 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-envoy\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.735449 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.735430 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-podinfo\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.740651 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.740630 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-token\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.741159 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.741128 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdh8w\" (UniqueName: \"kubernetes.io/projected/cecda5b5-849e-4c16-9258-9a2fd54555ac-kube-api-access-bdh8w\") pod \"e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:23.872516 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.872407 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 15:27:23.872516 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.872480 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" podUID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 15:27:23.908795 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:23.908756 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:24.104288 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:24.104254 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:24.204414 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:24.204379 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:24.219454 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:24.219416 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594"] Sep 11 15:27:24.227014 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:27:24.226977 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecda5b5_849e_4c16_9258_9a2fd54555ac.slice/crio-2153c58856e3e546ff9c9bd5fdf6772ec77a86a655c1781f14dd68e9822e7f21 WatchSource:0}: Error finding container 2153c58856e3e546ff9c9bd5fdf6772ec77a86a655c1781f14dd68e9822e7f21: Status 404 returned error can't find the container with id 2153c58856e3e546ff9c9bd5fdf6772ec77a86a655c1781f14dd68e9822e7f21 Sep 11 15:27:24.229321 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:24.229288 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:27:24.229378 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:24.229361 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:27:24.229426 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:24.229389 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:27:24.872812 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:24.872767 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 15:27:24.873318 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:24.872851 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" podUID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 15:27:24.901947 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:24.901917 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:25.003378 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:25.003348 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:25.132106 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:25.132010 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" event={"ID":"cecda5b5-849e-4c16-9258-9a2fd54555ac","Type":"ContainerStarted","Data":"40e4e0ec98db5b3eb8bb7fa465facf1bca9a4d05867166537e77c091378b28fd"} Sep 11 15:27:25.132106 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:25.132073 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" event={"ID":"cecda5b5-849e-4c16-9258-9a2fd54555ac","Type":"ContainerStarted","Data":"2153c58856e3e546ff9c9bd5fdf6772ec77a86a655c1781f14dd68e9822e7f21"} Sep 11 15:27:25.160810 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:25.160730 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" podStartSLOduration=2.160709237 podStartE2EDuration="2.160709237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:27:25.158319127 +0000 UTC m=+1139.390243574" watchObservedRunningTime="2026-09-11 15:27:25.160709237 +0000 UTC m=+1139.392633670" Sep 11 15:27:25.232497 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:25.232462 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:25.568764 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:25.567142 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:25.766865 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:25.766667 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:25.872704 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:25.872199 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 15:27:25.872704 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:25.872261 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" podUID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 15:27:25.909058 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:25.909023 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:25.914289 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:25.914269 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:26.139103 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:26.139009 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:26.140568 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:26.140544 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:26.872985 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:26.872950 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 15:27:26.873603 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:26.873554 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" podUID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 15:27:26.983522 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:26.983467 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:27.083363 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:27.083323 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:27.185275 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:27.185239 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:27.877157 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:27.877126 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:27:27.877523 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:27.877415 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:27:27.878487 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:27.878464 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:27:27.923786 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:27.923752 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:28.555136 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:28.555092 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:28.645141 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:28.645112 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:28.720075 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:28.720044 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:28.823036 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:28.822948 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:29.300272 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:29.300243 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:29.420754 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:29.420720 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:29.524542 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:29.524511 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:29.907900 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:29.907864 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq"] Sep 11 15:27:29.912678 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:29.912651 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:29.916388 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:29.916361 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-7ee57f-serving-cert\"" Sep 11 15:27:29.916552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:29.916407 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:27:29.916552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:29.916488 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-79rjx\"" Sep 11 15:27:29.916713 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:29.916653 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 15:27:29.918253 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:29.918234 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 15:27:29.924893 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:29.924871 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq"] Sep 11 15:27:29.943707 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:29.943687 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:30.013520 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.013382 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q9pd\" (UniqueName: \"kubernetes.io/projected/99bb113b-83ad-44c2-ab2d-3e35a5990220-kube-api-access-7q9pd\") pod \"maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:30.013920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.013889 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bb113b-83ad-44c2-ab2d-3e35a5990220-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:30.014164 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.014142 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bb113b-83ad-44c2-ab2d-3e35a5990220-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:30.014423 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.014402 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bb113b-83ad-44c2-ab2d-3e35a5990220-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:30.040648 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.040619 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:30.115110 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.115071 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bb113b-83ad-44c2-ab2d-3e35a5990220-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:30.115311 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.115126 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bb113b-83ad-44c2-ab2d-3e35a5990220-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:30.115311 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.115161 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q9pd\" (UniqueName: \"kubernetes.io/projected/99bb113b-83ad-44c2-ab2d-3e35a5990220-kube-api-access-7q9pd\") pod \"maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:30.115311 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.115216 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bb113b-83ad-44c2-ab2d-3e35a5990220-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:30.115311 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.115243 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:30.115850 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.115831 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bb113b-83ad-44c2-ab2d-3e35a5990220-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:30.117501 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.117477 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bb113b-83ad-44c2-ab2d-3e35a5990220-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:30.117658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.117635 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bb113b-83ad-44c2-ab2d-3e35a5990220-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:30.123934 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.123915 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q9pd\" (UniqueName: \"kubernetes.io/projected/99bb113b-83ad-44c2-ab2d-3e35a5990220-kube-api-access-7q9pd\") pod \"maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:30.225720 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.225643 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:30.506462 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.506435 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq"] Sep 11 15:27:30.508478 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:27:30.508453 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bb113b_83ad_44c2_ab2d_3e35a5990220.slice/crio-1f130f2b992ef9785ecf46bd7c823676a47f05a9550aa948576788e568c397c7 WatchSource:0}: Error finding container 1f130f2b992ef9785ecf46bd7c823676a47f05a9550aa948576788e568c397c7: Status 404 returned error can't find the container with id 1f130f2b992ef9785ecf46bd7c823676a47f05a9550aa948576788e568c397c7 Sep 11 15:27:30.640235 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.640200 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:30.736283 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:30.736250 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:31.093433 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:31.093342 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:31.170188 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:31.170153 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" event={"ID":"99bb113b-83ad-44c2-ab2d-3e35a5990220","Type":"ContainerStarted","Data":"4f30ef15bf830720118eefdd6464c3c224d693e4ada46dd1e974f809edc1001c"} Sep 11 15:27:31.170368 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:31.170243 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" event={"ID":"99bb113b-83ad-44c2-ab2d-3e35a5990220","Type":"ContainerStarted","Data":"1f130f2b992ef9785ecf46bd7c823676a47f05a9550aa948576788e568c397c7"} Sep 11 15:27:31.170420 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:31.170367 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:31.192050 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:31.191998 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" podStartSLOduration=1.874477881 podStartE2EDuration="2.19198407s" totalImagesPullingTime="317.506189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:27:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:27:31.191025497 +0000 UTC m=+1145.422949929" watchObservedRunningTime="2026-09-11 15:27:31.19198407 +0000 UTC m=+1145.423908501" Sep 11 15:27:31.853685 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:31.853654 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:32.728537 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:32.728505 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:33.364538 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:33.364510 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:35.488190 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:35.488153 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:27:37.181448 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:37.181417 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:27:47.259165 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:47.259131 2558 generic.go:356] "Generic (PLEG): container finished" podID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerID="40e4e0ec98db5b3eb8bb7fa465facf1bca9a4d05867166537e77c091378b28fd" exitCode=0 Sep 11 15:27:47.259575 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:47.259204 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" event={"ID":"cecda5b5-849e-4c16-9258-9a2fd54555ac","Type":"ContainerDied","Data":"40e4e0ec98db5b3eb8bb7fa465facf1bca9a4d05867166537e77c091378b28fd"} Sep 11 15:27:47.259575 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:47.259545 2558 scope.go:117] "RemoveContainer" containerID="40e4e0ec98db5b3eb8bb7fa465facf1bca9a4d05867166537e77c091378b28fd" Sep 11 15:27:47.260602 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:47.260564 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:27:47.260707 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:47.260636 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:27:47.260707 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:47.260668 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:27:48.274349 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:48.274312 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" event={"ID":"cecda5b5-849e-4c16-9258-9a2fd54555ac","Type":"ContainerStarted","Data":"0f4b673f656c4072e9b138b0d8563546f30ba0bd6bc36f4f0ab473d04add38c5"} Sep 11 15:27:48.909483 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:48.909399 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:48.909637 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:48.909621 2558 patch_prober.go:28] interesting pod/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 11 15:27:48.909680 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:48.909664 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 11 15:27:49.828606 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:49.828567 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:49.909522 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:49.909490 2558 patch_prober.go:28] interesting pod/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 11 15:27:49.909690 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:49.909552 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 11 15:27:50.909282 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:50.909244 2558 patch_prober.go:28] interesting pod/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 11 15:27:50.909667 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:50.909314 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 11 15:27:51.471808 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:51.471771 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:51.909277 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:51.909244 2558 patch_prober.go:28] interesting pod/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 11 15:27:51.909440 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:51.909322 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 11 15:27:52.085624 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:52.085595 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:52.486834 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:52.486796 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:52.909397 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:52.909368 2558 patch_prober.go:28] interesting pod/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 11 15:27:52.909565 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:52.909419 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 11 15:27:53.176549 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:53.176456 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:53.818245 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:53.818212 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:53.913505 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:53.913473 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:53.913930 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:53.913777 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:53.914346 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:53.914332 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:27:54.619475 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:54.619441 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:54.713349 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:54.713322 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:55.438183 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:55.438152 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:56.143984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:56.143951 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:56.825170 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:56.825134 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:57.619765 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:57.619708 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:58.319193 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:58.319158 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:58.573102 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:58.573011 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:27:58.597161 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:27:58.597123 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:28:07.388874 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.388839 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:07.673387 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.673354 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m"] Sep 11 15:28:07.677698 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.677677 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.680139 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.680122 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-dcdcc9ba-openshift-default-dockercfg-c8fmv\"" Sep 11 15:28:07.689418 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.689394 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m"] Sep 11 15:28:07.701515 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.701488 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:07.765267 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.765230 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-podinfo\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.765513 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.765493 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-data\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.765701 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.765686 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-workload-certs\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.765830 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.765815 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-envoy\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.765937 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.765924 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istiod-ca-cert\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.766038 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.766026 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-token\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.766155 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.766143 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-credential-socket\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.766252 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.766238 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4l2\" (UniqueName: \"kubernetes.io/projected/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-kube-api-access-cr4l2\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.766389 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.766377 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-workload-socket\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.816541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.816510 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:07.868145 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.868097 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-workload-certs\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.868345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.868154 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-envoy\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.868345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.868180 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istiod-ca-cert\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.868345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.868213 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-token\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.868345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.868256 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-credential-socket\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.868345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.868288 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4l2\" (UniqueName: \"kubernetes.io/projected/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-kube-api-access-cr4l2\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.868639 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.868365 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-workload-socket\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.868639 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.868424 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-podinfo\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.868639 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.868462 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-data\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.869022 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.868993 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-data\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.869508 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.869483 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-workload-certs\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.869619 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.869583 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-workload-socket\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.870000 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.869953 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istiod-ca-cert\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.870303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.870283 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-credential-socket\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.872234 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.872215 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-envoy\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.872474 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.872456 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-podinfo\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.877075 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.877052 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-token\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.877174 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.877152 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4l2\" (UniqueName: \"kubernetes.io/projected/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-kube-api-access-cr4l2\") pod \"e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:07.989803 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:07.989681 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:08.265115 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:08.265089 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m"] Sep 11 15:28:08.266687 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:28:08.266658 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e87a60f_0aa2_4c8c_97be_cefef2e0e84c.slice/crio-e2f8f570fdbadb4c03676799672b5f422f99e111875d6ce97197dc5f9d992724 WatchSource:0}: Error finding container e2f8f570fdbadb4c03676799672b5f422f99e111875d6ce97197dc5f9d992724: Status 404 returned error can't find the container with id e2f8f570fdbadb4c03676799672b5f422f99e111875d6ce97197dc5f9d992724 Sep 11 15:28:08.269358 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:08.269324 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:28:08.269457 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:08.269400 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:28:08.269457 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:08.269446 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:28:08.395543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:08.395500 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" event={"ID":"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c","Type":"ContainerStarted","Data":"2f21b1290930fc3a44cf008e526be8e1bc1595ed052af22d0bf04388cdc06549"} Sep 11 15:28:08.395932 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:08.395559 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" event={"ID":"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c","Type":"ContainerStarted","Data":"e2f8f570fdbadb4c03676799672b5f422f99e111875d6ce97197dc5f9d992724"} Sep 11 15:28:08.417087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:08.417039 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" podStartSLOduration=1.417022998 podStartE2EDuration="1.417022998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:28:08.415506826 +0000 UTC m=+1182.647431291" watchObservedRunningTime="2026-09-11 15:28:08.417022998 +0000 UTC m=+1182.648947429" Sep 11 15:28:08.990386 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:08.990356 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:09.620920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:09.620892 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:09.996005 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:09.995979 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:10.407249 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:10.407223 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:10.408338 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:10.408319 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:10.416714 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:10.416663 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:10.511938 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:10.511906 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:11.222538 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:11.222499 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:12.143158 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:12.143125 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:12.817413 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:12.817381 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:13.581825 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.581781 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt"] Sep 11 15:28:13.586177 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.586147 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:13.588936 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.588915 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-dcdcc9ba-serving-cert\"" Sep 11 15:28:13.596397 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.596374 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt"] Sep 11 15:28:13.614044 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.614023 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:13.621887 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.621866 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c88e9a-d436-4234-bb58-6aadc35583b2-openshift-service-ca\") pod \"maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:13.621972 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.621920 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-metrics-certs\") pod \"maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:13.622019 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.621994 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-maas-api-tls\") pod \"maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:13.622076 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.622062 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxcmb\" (UniqueName: \"kubernetes.io/projected/90c88e9a-d436-4234-bb58-6aadc35583b2-kube-api-access-gxcmb\") pod \"maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:13.716618 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.716586 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:13.722911 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.722887 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxcmb\" (UniqueName: \"kubernetes.io/projected/90c88e9a-d436-4234-bb58-6aadc35583b2-kube-api-access-gxcmb\") pod \"maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:13.723007 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.722944 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c88e9a-d436-4234-bb58-6aadc35583b2-openshift-service-ca\") pod \"maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:13.723007 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.722991 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-metrics-certs\") pod \"maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:13.723089 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.723037 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-maas-api-tls\") pod \"maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:13.723183 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:28:13.723168 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-serving-cert: secret "maas-api-e2e-mt-dcdcc9ba-serving-cert" not found Sep 11 15:28:13.723256 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:28:13.723245 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-maas-api-tls podName:90c88e9a-d436-4234-bb58-6aadc35583b2 nodeName:}" failed. No retries permitted until 2026-09-11 15:28:14.223226223 +0000 UTC m=+1188.455150634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-maas-api-tls") pod "maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" (UID: "90c88e9a-d436-4234-bb58-6aadc35583b2") : secret "maas-api-e2e-mt-dcdcc9ba-serving-cert" not found Sep 11 15:28:13.723664 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.723646 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c88e9a-d436-4234-bb58-6aadc35583b2-openshift-service-ca\") pod \"maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:13.725471 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.725454 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-metrics-certs\") pod \"maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:13.731004 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.730980 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxcmb\" (UniqueName: \"kubernetes.io/projected/90c88e9a-d436-4234-bb58-6aadc35583b2-kube-api-access-gxcmb\") pod \"maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:13.816655 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:13.816601 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:14.071848 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:14.071817 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:14.189377 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:14.189343 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:14.228797 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:14.228767 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-maas-api-tls\") pod \"maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:14.231159 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:14.231129 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-maas-api-tls\") pod \"maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:14.499885 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:14.499845 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:14.773379 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:14.773353 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt"] Sep 11 15:28:14.774444 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:28:14.774413 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c88e9a_d436_4234_bb58_6aadc35583b2.slice/crio-dce2ecb4500ab20926a758594ef762ae72cae73536ff69c9aae8efa7a5beb1ec WatchSource:0}: Error finding container dce2ecb4500ab20926a758594ef762ae72cae73536ff69c9aae8efa7a5beb1ec: Status 404 returned error can't find the container with id dce2ecb4500ab20926a758594ef762ae72cae73536ff69c9aae8efa7a5beb1ec Sep 11 15:28:15.435976 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:15.435947 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" event={"ID":"90c88e9a-d436-4234-bb58-6aadc35583b2","Type":"ContainerStarted","Data":"a6e688f2473713fd4cf9acbbe0e2515cd1f10a599fc72d0e620f256daeec9a67"} Sep 11 15:28:15.436332 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:15.436018 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:15.436332 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:15.436044 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" event={"ID":"90c88e9a-d436-4234-bb58-6aadc35583b2","Type":"ContainerStarted","Data":"dce2ecb4500ab20926a758594ef762ae72cae73536ff69c9aae8efa7a5beb1ec"} Sep 11 15:28:15.454393 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:15.454339 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" podStartSLOduration=1.9069115810000001 podStartE2EDuration="2.454323345s" totalImagesPullingTime="547.411764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:28:15.450767683 +0000 UTC m=+1189.682692106" watchObservedRunningTime="2026-09-11 15:28:15.454323345 +0000 UTC m=+1189.686247780" Sep 11 15:28:18.186949 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:18.186910 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:28:21.447661 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:21.447634 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:28:26.008612 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:26.008575 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:26.393828 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:26.393794 2558 scope.go:117] "RemoveContainer" containerID="62aebcb075ebc4571a03730fd6a2ea3db38206e02fc30e2da3deff4cb0140bd8" Sep 11 15:28:26.449885 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:26.449837 2558 patch_prober.go:28] interesting pod/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 15:28:26.450054 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:26.449892 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 15:28:28.525164 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:28.525131 2558 generic.go:356] "Generic (PLEG): container finished" podID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerID="2f21b1290930fc3a44cf008e526be8e1bc1595ed052af22d0bf04388cdc06549" exitCode=0 Sep 11 15:28:28.525598 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:28.525208 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" event={"ID":"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c","Type":"ContainerDied","Data":"2f21b1290930fc3a44cf008e526be8e1bc1595ed052af22d0bf04388cdc06549"} Sep 11 15:28:28.525647 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:28.525623 2558 scope.go:117] "RemoveContainer" containerID="2f21b1290930fc3a44cf008e526be8e1bc1595ed052af22d0bf04388cdc06549" Sep 11 15:28:28.526817 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:28.526771 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:28:28.526934 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:28.526882 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:28:28.526934 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:28.526929 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:28:29.533137 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:29.533101 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" event={"ID":"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c","Type":"ContainerStarted","Data":"7c32643f9f5483b632c4dd0b9729e23dd285fc891c9c3d2a82ace34a2a354bcc"} Sep 11 15:28:29.990876 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:29.990839 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:29.991087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:29.991064 2558 patch_prober.go:28] interesting pod/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 11 15:28:29.991133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:29.991113 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 11 15:28:30.520000 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:30.519965 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:30.991012 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:30.990975 2558 patch_prober.go:28] interesting pod/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 11 15:28:30.991502 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:30.991033 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 11 15:28:31.990074 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:31.990038 2558 patch_prober.go:28] interesting pod/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 11 15:28:31.990258 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:31.990135 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 11 15:28:32.154647 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:32.154588 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:32.604088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:32.604046 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:28:32.990812 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:32.990768 2558 patch_prober.go:28] interesting pod/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 11 15:28:32.990986 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:32.990832 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 11 15:28:33.990251 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:33.990206 2558 patch_prober.go:28] interesting pod/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Sep 11 15:28:33.990642 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:33.990282 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Sep 11 15:28:34.553023 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:34.552983 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:34.993867 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:34.993839 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:34.994288 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:34.994186 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:34.994782 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:34.994762 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:28:35.935112 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:35.935077 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:36.716825 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:36.716795 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:37.434421 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:37.434393 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:38.329493 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:38.329456 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:38.984664 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:38.984634 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:39.657334 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:39.657300 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:40.496996 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:40.496963 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:28:49.692318 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:49.692282 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:28:52.609580 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:28:52.609546 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:07.889960 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:07.889929 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:29:21.120232 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:21.120128 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:22.676852 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:22.676820 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:23.181437 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:23.181403 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:23.904617 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:23.904581 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:24.667674 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:24.667637 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:25.979206 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:25.979168 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:29:26.419576 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:26.419534 2558 scope.go:117] "RemoveContainer" containerID="05e257c89dc018baa52b70a199014ee695012bf2b689450593ca4aea1fa5a583" Sep 11 15:29:35.725416 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:35.725372 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt"] Sep 11 15:29:35.725842 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:35.725638 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" podUID="90c88e9a-d436-4234-bb58-6aadc35583b2" containerName="maas-api" containerID="cri-o://a6e688f2473713fd4cf9acbbe0e2515cd1f10a599fc72d0e620f256daeec9a67" gracePeriod=30 Sep 11 15:29:35.752350 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:35.752314 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:35.815892 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:35.815854 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:35.889895 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:35.889849 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:35.924380 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:35.924347 2558 generic.go:356] "Generic (PLEG): container finished" podID="90c88e9a-d436-4234-bb58-6aadc35583b2" containerID="a6e688f2473713fd4cf9acbbe0e2515cd1f10a599fc72d0e620f256daeec9a67" exitCode=0 Sep 11 15:29:35.924560 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:35.924417 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" event={"ID":"90c88e9a-d436-4234-bb58-6aadc35583b2","Type":"ContainerDied","Data":"a6e688f2473713fd4cf9acbbe0e2515cd1f10a599fc72d0e620f256daeec9a67"} Sep 11 15:29:35.958407 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:35.958363 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:35.990294 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:35.990215 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:29:36.022155 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.022126 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:36.090046 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.090019 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxcmb\" (UniqueName: \"kubernetes.io/projected/90c88e9a-d436-4234-bb58-6aadc35583b2-kube-api-access-gxcmb\") pod \"90c88e9a-d436-4234-bb58-6aadc35583b2\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " Sep 11 15:29:36.090211 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.090132 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c88e9a-d436-4234-bb58-6aadc35583b2-openshift-service-ca\") pod \"90c88e9a-d436-4234-bb58-6aadc35583b2\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " Sep 11 15:29:36.090211 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.090153 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-maas-api-tls\") pod \"90c88e9a-d436-4234-bb58-6aadc35583b2\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " Sep 11 15:29:36.090211 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.090174 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-metrics-certs\") pod \"90c88e9a-d436-4234-bb58-6aadc35583b2\" (UID: \"90c88e9a-d436-4234-bb58-6aadc35583b2\") " Sep 11 15:29:36.090555 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.090528 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c88e9a-d436-4234-bb58-6aadc35583b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90c88e9a-d436-4234-bb58-6aadc35583b2" (UID: "90c88e9a-d436-4234-bb58-6aadc35583b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:29:36.092329 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.092303 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90c88e9a-d436-4234-bb58-6aadc35583b2" (UID: "90c88e9a-d436-4234-bb58-6aadc35583b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:29:36.092329 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.092320 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c88e9a-d436-4234-bb58-6aadc35583b2-kube-api-access-gxcmb" (OuterVolumeSpecName: "kube-api-access-gxcmb") pod "90c88e9a-d436-4234-bb58-6aadc35583b2" (UID: "90c88e9a-d436-4234-bb58-6aadc35583b2"). InnerVolumeSpecName "kube-api-access-gxcmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:29:36.092329 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.092318 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "90c88e9a-d436-4234-bb58-6aadc35583b2" (UID: "90c88e9a-d436-4234-bb58-6aadc35583b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:29:36.191892 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.191856 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90c88e9a-d436-4234-bb58-6aadc35583b2-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:29:36.191892 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.191885 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:29:36.191892 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.191894 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90c88e9a-d436-4234-bb58-6aadc35583b2-metrics-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:29:36.191892 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.191902 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxcmb\" (UniqueName: \"kubernetes.io/projected/90c88e9a-d436-4234-bb58-6aadc35583b2-kube-api-access-gxcmb\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:29:36.504618 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.504582 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:36.931079 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.931043 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" event={"ID":"90c88e9a-d436-4234-bb58-6aadc35583b2","Type":"ContainerDied","Data":"dce2ecb4500ab20926a758594ef762ae72cae73536ff69c9aae8efa7a5beb1ec"} Sep 11 15:29:36.931474 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.931128 2558 scope.go:117] "RemoveContainer" containerID="a6e688f2473713fd4cf9acbbe0e2515cd1f10a599fc72d0e620f256daeec9a67" Sep 11 15:29:36.931474 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.931053 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt" Sep 11 15:29:36.947002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.946979 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt"] Sep 11 15:29:36.950610 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:36.950586 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dcdcc9ba-bbf545969-m5vnt"] Sep 11 15:29:37.209203 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:37.209122 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:37.917450 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:37.917423 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:38.264188 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:38.264108 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90c88e9a-d436-4234-bb58-6aadc35583b2" path="/var/lib/kubelet/pods/90c88e9a-d436-4234-bb58-6aadc35583b2/volumes" Sep 11 15:29:38.729873 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:38.729842 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:39.361719 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:39.361687 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:39.462065 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:39.462032 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:40.690751 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:40.690715 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:29:44.437129 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:44.437097 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:29:56.984906 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:29:56.984870 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:30:00.148782 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.148723 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819010-jph5p"] Sep 11 15:30:00.149359 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.149140 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c88e9a-d436-4234-bb58-6aadc35583b2" containerName="maas-api" Sep 11 15:30:00.149359 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.149152 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c88e9a-d436-4234-bb58-6aadc35583b2" containerName="maas-api" Sep 11 15:30:00.149359 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.149214 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c88e9a-d436-4234-bb58-6aadc35583b2" containerName="maas-api" Sep 11 15:30:00.152421 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.152391 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt"] Sep 11 15:30:00.152550 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.152532 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819010-jph5p" Sep 11 15:30:00.154657 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.154634 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:30:00.156068 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.156045 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt" Sep 11 15:30:00.162523 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.162486 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819010-jph5p"] Sep 11 15:30:00.166830 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.166811 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819010-jph5p" Sep 11 15:30:00.173461 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.173236 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt" Sep 11 15:30:00.178512 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.178479 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt"] Sep 11 15:30:00.455878 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.455852 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819010-jph5p"] Sep 11 15:30:00.458946 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:30:00.458917 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3f8956_5bb1_4ef3_a3f5_aed47b6fcbe2.slice/crio-1f548aa5ab122bbf295b38e7ef2fc258a7696379ec6b75df66efa1e759eb638c WatchSource:0}: Error finding container 1f548aa5ab122bbf295b38e7ef2fc258a7696379ec6b75df66efa1e759eb638c: Status 404 returned error can't find the container with id 1f548aa5ab122bbf295b38e7ef2fc258a7696379ec6b75df66efa1e759eb638c Sep 11 15:30:00.460922 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.460898 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt"] Sep 11 15:30:00.460922 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:00.460911 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 15:30:00.461255 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:30:00.461235 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2324a3a_892e_45c5_b20c_a4648e51f0ec.slice/crio-65f2f0a42dd3e671e1db0f4f44c4a5cf46a9e4fcdac17711a9d3c92653e23bcc WatchSource:0}: Error finding container 65f2f0a42dd3e671e1db0f4f44c4a5cf46a9e4fcdac17711a9d3c92653e23bcc: Status 404 returned error can't find the container with id 65f2f0a42dd3e671e1db0f4f44c4a5cf46a9e4fcdac17711a9d3c92653e23bcc Sep 11 15:30:01.064123 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:01.064082 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819010-jph5p" event={"ID":"1f3f8956-5bb1-4ef3-a3f5-aed47b6fcbe2","Type":"ContainerStarted","Data":"1f548aa5ab122bbf295b38e7ef2fc258a7696379ec6b75df66efa1e759eb638c"} Sep 11 15:30:01.065313 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:01.065282 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt" event={"ID":"c2324a3a-892e-45c5-b20c-a4648e51f0ec","Type":"ContainerStarted","Data":"65f2f0a42dd3e671e1db0f4f44c4a5cf46a9e4fcdac17711a9d3c92653e23bcc"} Sep 11 15:30:04.085800 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:04.085771 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819010-jph5p" event={"ID":"1f3f8956-5bb1-4ef3-a3f5-aed47b6fcbe2","Type":"ContainerStarted","Data":"4ea6e73c52f9da18696d5cfa29394c8fed8b20f745b78d6e5e34748a27335396"} Sep 11 15:30:04.087379 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:04.087357 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt" event={"ID":"c2324a3a-892e-45c5-b20c-a4648e51f0ec","Type":"ContainerStarted","Data":"d57ef6f7793055d012d61986b40cdbc21ac64757553bd7cd4889469be4441cfe"} Sep 11 15:30:04.102345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:04.102302 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819010-jph5p" podStartSLOduration=0.745818881 podStartE2EDuration="4.102288552s" totalImagesPullingTime="3.356469671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:30:04.09945129 +0000 UTC m=+1298.331375722" watchObservedRunningTime="2026-09-11 15:30:04.102288552 +0000 UTC m=+1298.334212983" Sep 11 15:30:04.118175 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:04.118132 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt" podStartSLOduration=0.767715262 podStartE2EDuration="4.118118592s" totalImagesPullingTime="3.35040333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:30:04.116016585 +0000 UTC m=+1298.347941016" watchObservedRunningTime="2026-09-11 15:30:04.118118592 +0000 UTC m=+1298.350043023" Sep 11 15:30:05.094420 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:05.094332 2558 generic.go:356] "Generic (PLEG): container finished" podID="c2324a3a-892e-45c5-b20c-a4648e51f0ec" containerID="d57ef6f7793055d012d61986b40cdbc21ac64757553bd7cd4889469be4441cfe" exitCode=0 Sep 11 15:30:05.094869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:05.094405 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt" event={"ID":"c2324a3a-892e-45c5-b20c-a4648e51f0ec","Type":"ContainerDied","Data":"d57ef6f7793055d012d61986b40cdbc21ac64757553bd7cd4889469be4441cfe"} Sep 11 15:30:05.096312 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:05.096288 2558 generic.go:356] "Generic (PLEG): container finished" podID="1f3f8956-5bb1-4ef3-a3f5-aed47b6fcbe2" containerID="4ea6e73c52f9da18696d5cfa29394c8fed8b20f745b78d6e5e34748a27335396" exitCode=0 Sep 11 15:30:05.096424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:05.096331 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819010-jph5p" event={"ID":"1f3f8956-5bb1-4ef3-a3f5-aed47b6fcbe2","Type":"ContainerDied","Data":"4ea6e73c52f9da18696d5cfa29394c8fed8b20f745b78d6e5e34748a27335396"} Sep 11 15:30:06.225217 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:06.225184 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819010-jph5p" Sep 11 15:30:06.280613 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:06.280590 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt" Sep 11 15:30:07.036896 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:07.036863 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:07.108413 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:07.108383 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt" Sep 11 15:30:07.108574 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:07.108412 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt" event={"ID":"c2324a3a-892e-45c5-b20c-a4648e51f0ec","Type":"ContainerDied","Data":"65f2f0a42dd3e671e1db0f4f44c4a5cf46a9e4fcdac17711a9d3c92653e23bcc"} Sep 11 15:30:07.108574 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:07.108479 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65f2f0a42dd3e671e1db0f4f44c4a5cf46a9e4fcdac17711a9d3c92653e23bcc" Sep 11 15:30:07.109855 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:07.109828 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819010-jph5p" event={"ID":"1f3f8956-5bb1-4ef3-a3f5-aed47b6fcbe2","Type":"ContainerDied","Data":"1f548aa5ab122bbf295b38e7ef2fc258a7696379ec6b75df66efa1e759eb638c"} Sep 11 15:30:07.109961 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:07.109860 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819010-jph5p" Sep 11 15:30:07.109961 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:07.109868 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f548aa5ab122bbf295b38e7ef2fc258a7696379ec6b75df66efa1e759eb638c" Sep 11 15:30:07.137797 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:07.137775 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:12.783063 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:12.783023 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:30:20.614623 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:20.614587 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:21.626786 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:21.626671 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:26.444190 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:26.444157 2558 scope.go:117] "RemoveContainer" containerID="1b04af8b4472b3a603615432e587450c36d4c88a7e273aab9f4e9f98d86d90ca" Sep 11 15:30:26.997169 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:26.997136 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:27.287134 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:27.287055 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:30:27.288553 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:27.288531 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:27.388858 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:27.388827 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:27.493464 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:27.493429 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m"] Sep 11 15:30:27.493946 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:27.493695 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerName="istio-proxy" containerID="cri-o://7c32643f9f5483b632c4dd0b9729e23dd285fc891c9c3d2a82ace34a2a354bcc" gracePeriod=30 Sep 11 15:30:27.517374 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:27.517349 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:28.099890 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:28.099861 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:28.753078 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:28.753042 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:28.852203 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:28.852169 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:28.955573 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:28.955544 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:29.521217 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:29.521185 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:29.617237 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:29.617202 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:29.726281 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:29.726250 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:30.974701 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:30.974660 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:32.749588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.749563 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:30:32.807383 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.807305 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:32.809448 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.809426 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istiod-ca-cert\") pod \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " Sep 11 15:30:32.809519 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.809471 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-envoy\") pod \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " Sep 11 15:30:32.809561 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.809525 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-workload-socket\") pod \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " Sep 11 15:30:32.809561 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.809545 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-token\") pod \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " Sep 11 15:30:32.809654 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.809562 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-credential-socket\") pod \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " Sep 11 15:30:32.809654 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.809588 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr4l2\" (UniqueName: \"kubernetes.io/projected/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-kube-api-access-cr4l2\") pod \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " Sep 11 15:30:32.809654 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.809638 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-data\") pod \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " Sep 11 15:30:32.809851 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.809683 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-podinfo\") pod \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " Sep 11 15:30:32.809851 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.809730 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-workload-certs\") pod \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\" (UID: \"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c\") " Sep 11 15:30:32.809961 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.809860 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" (UID: "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:30:32.810040 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.809990 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-data" (OuterVolumeSpecName: "istio-data") pod "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" (UID: "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:30:32.810104 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.810057 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" (UID: "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:30:32.810157 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.810110 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" (UID: "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:30:32.810217 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.810165 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:30:32.810217 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.810186 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:30:32.810217 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.810201 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:30:32.810330 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.810227 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" (UID: "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:30:32.811795 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.811768 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" (UID: "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:30:32.811918 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.811875 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" (UID: "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:30:32.812207 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.812184 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" (UID: "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:30:32.812269 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.812189 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-kube-api-access-cr4l2" (OuterVolumeSpecName: "kube-api-access-cr4l2") pod "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" (UID: "4e87a60f-0aa2-4c8c-97be-cefef2e0e84c"). InnerVolumeSpecName "kube-api-access-cr4l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:30:32.910817 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.910790 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:30:32.910817 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.910812 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:30:32.910817 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.910821 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:30:32.911003 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.910830 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:30:32.911003 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.910838 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:30:32.911003 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:32.910846 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr4l2\" (UniqueName: \"kubernetes.io/projected/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c-kube-api-access-cr4l2\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:30:33.253088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:33.253050 2558 generic.go:356] "Generic (PLEG): container finished" podID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerID="7c32643f9f5483b632c4dd0b9729e23dd285fc891c9c3d2a82ace34a2a354bcc" exitCode=0 Sep 11 15:30:33.253255 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:33.253108 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" Sep 11 15:30:33.253255 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:33.253138 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" event={"ID":"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c","Type":"ContainerDied","Data":"7c32643f9f5483b632c4dd0b9729e23dd285fc891c9c3d2a82ace34a2a354bcc"} Sep 11 15:30:33.253255 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:33.253195 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m" event={"ID":"4e87a60f-0aa2-4c8c-97be-cefef2e0e84c","Type":"ContainerDied","Data":"e2f8f570fdbadb4c03676799672b5f422f99e111875d6ce97197dc5f9d992724"} Sep 11 15:30:33.253255 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:33.253219 2558 scope.go:117] "RemoveContainer" containerID="7c32643f9f5483b632c4dd0b9729e23dd285fc891c9c3d2a82ace34a2a354bcc" Sep 11 15:30:33.263270 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:33.263254 2558 scope.go:117] "RemoveContainer" containerID="2f21b1290930fc3a44cf008e526be8e1bc1595ed052af22d0bf04388cdc06549" Sep 11 15:30:33.271226 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:33.271210 2558 scope.go:117] "RemoveContainer" containerID="7c32643f9f5483b632c4dd0b9729e23dd285fc891c9c3d2a82ace34a2a354bcc" Sep 11 15:30:33.271501 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:30:33.271479 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c32643f9f5483b632c4dd0b9729e23dd285fc891c9c3d2a82ace34a2a354bcc\": container with ID starting with 7c32643f9f5483b632c4dd0b9729e23dd285fc891c9c3d2a82ace34a2a354bcc not found: ID does not exist" containerID="7c32643f9f5483b632c4dd0b9729e23dd285fc891c9c3d2a82ace34a2a354bcc" Sep 11 15:30:33.271541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:33.271513 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c32643f9f5483b632c4dd0b9729e23dd285fc891c9c3d2a82ace34a2a354bcc"} err="failed to get container status \"7c32643f9f5483b632c4dd0b9729e23dd285fc891c9c3d2a82ace34a2a354bcc\": rpc error: code = NotFound desc = could not find container \"7c32643f9f5483b632c4dd0b9729e23dd285fc891c9c3d2a82ace34a2a354bcc\": container with ID starting with 7c32643f9f5483b632c4dd0b9729e23dd285fc891c9c3d2a82ace34a2a354bcc not found: ID does not exist" Sep 11 15:30:33.271541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:33.271536 2558 scope.go:117] "RemoveContainer" containerID="2f21b1290930fc3a44cf008e526be8e1bc1595ed052af22d0bf04388cdc06549" Sep 11 15:30:33.271822 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:30:33.271802 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f21b1290930fc3a44cf008e526be8e1bc1595ed052af22d0bf04388cdc06549\": container with ID starting with 2f21b1290930fc3a44cf008e526be8e1bc1595ed052af22d0bf04388cdc06549 not found: ID does not exist" containerID="2f21b1290930fc3a44cf008e526be8e1bc1595ed052af22d0bf04388cdc06549" Sep 11 15:30:33.271869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:33.271827 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f21b1290930fc3a44cf008e526be8e1bc1595ed052af22d0bf04388cdc06549"} err="failed to get container status \"2f21b1290930fc3a44cf008e526be8e1bc1595ed052af22d0bf04388cdc06549\": rpc error: code = NotFound desc = could not find container \"2f21b1290930fc3a44cf008e526be8e1bc1595ed052af22d0bf04388cdc06549\": container with ID starting with 2f21b1290930fc3a44cf008e526be8e1bc1595ed052af22d0bf04388cdc06549 not found: ID does not exist" Sep 11 15:30:33.276802 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:33.276779 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m"] Sep 11 15:30:33.283860 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:33.283842 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-dcdcc9ba-openshift-default-5bd668c44d-c2m8m"] Sep 11 15:30:34.264273 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:34.264242 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" path="/var/lib/kubelet/pods/4e87a60f-0aa2-4c8c-97be-cefef2e0e84c/volumes" Sep 11 15:30:37.043826 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:37.043789 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:37.110954 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:37.110910 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:37.686570 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:37.686538 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:41.794489 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:41.794453 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:30:43.471132 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:43.471096 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:43.551376 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:43.551343 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:43.659831 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:43.659801 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:45.016296 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:45.016268 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:48.120051 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:48.120015 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:54.145967 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:54.145884 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:54.359088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:54.359050 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:30:55.192855 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:30:55.192820 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:31:15.222940 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:15.222909 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:15.323608 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:15.323579 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:19.588243 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:19.588209 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:31:20.913550 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:20.913517 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:21.013809 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.013780 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:21.287428 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.287340 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:21.576170 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.576087 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs"] Sep 11 15:31:21.576630 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.576611 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerName="istio-proxy" Sep 11 15:31:21.576630 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.576630 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerName="istio-proxy" Sep 11 15:31:21.576787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.576649 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerName="istio-proxy" Sep 11 15:31:21.576787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.576658 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerName="istio-proxy" Sep 11 15:31:21.576787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.576677 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f3f8956-5bb1-4ef3-a3f5-aed47b6fcbe2" containerName="cleanup" Sep 11 15:31:21.576787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.576685 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f3f8956-5bb1-4ef3-a3f5-aed47b6fcbe2" containerName="cleanup" Sep 11 15:31:21.576787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.576715 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2324a3a-892e-45c5-b20c-a4648e51f0ec" containerName="cleanup" Sep 11 15:31:21.576787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.576724 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2324a3a-892e-45c5-b20c-a4648e51f0ec" containerName="cleanup" Sep 11 15:31:21.576966 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.576816 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2324a3a-892e-45c5-b20c-a4648e51f0ec" containerName="cleanup" Sep 11 15:31:21.576966 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.576830 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f3f8956-5bb1-4ef3-a3f5-aed47b6fcbe2" containerName="cleanup" Sep 11 15:31:21.576966 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.576840 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerName="istio-proxy" Sep 11 15:31:21.580297 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.580277 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.582543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.582517 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-808212-openshift-default-dockercfg-4zpf4\"" Sep 11 15:31:21.588832 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.588811 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs"] Sep 11 15:31:21.593862 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.593831 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:21.636840 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.636811 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-istio-data\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.636954 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.636862 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-workload-socket\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.636954 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.636941 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-credential-socket\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.637084 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.637059 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-workload-certs\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.637144 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.637116 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11322aa2-1920-420d-9d97-034446a0337d-istiod-ca-cert\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.637203 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.637148 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lppp\" (UniqueName: \"kubernetes.io/projected/11322aa2-1920-420d-9d97-034446a0337d-kube-api-access-5lppp\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.637203 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.637181 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11322aa2-1920-420d-9d97-034446a0337d-istio-podinfo\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.637282 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.637210 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11322aa2-1920-420d-9d97-034446a0337d-istio-token\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.637282 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.637255 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-istio-envoy\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.721832 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.721794 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:21.738547 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.738507 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-workload-certs\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.738716 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.738555 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11322aa2-1920-420d-9d97-034446a0337d-istiod-ca-cert\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.738716 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.738631 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lppp\" (UniqueName: \"kubernetes.io/projected/11322aa2-1920-420d-9d97-034446a0337d-kube-api-access-5lppp\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.738716 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.738654 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11322aa2-1920-420d-9d97-034446a0337d-istio-podinfo\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.738716 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.738673 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11322aa2-1920-420d-9d97-034446a0337d-istio-token\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.738716 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.738714 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-istio-envoy\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.739006 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.738770 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-istio-data\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.739006 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.738805 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-workload-socket\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.739006 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.738918 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-credential-socket\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.739247 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.739195 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-workload-certs\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.739247 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.739234 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-workload-socket\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.739342 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.739273 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-istio-data\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.739342 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.739319 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-credential-socket\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.739342 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.739330 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11322aa2-1920-420d-9d97-034446a0337d-istiod-ca-cert\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.741086 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.741064 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-istio-envoy\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.741184 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.741143 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11322aa2-1920-420d-9d97-034446a0337d-istio-podinfo\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.751239 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.751216 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lppp\" (UniqueName: \"kubernetes.io/projected/11322aa2-1920-420d-9d97-034446a0337d-kube-api-access-5lppp\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.754270 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.754248 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11322aa2-1920-420d-9d97-034446a0337d-istio-token\") pod \"e2e-api-a-808212-openshift-default-66845f8df4-mj7bs\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:21.892061 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:21.892038 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:22.176109 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:22.172786 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs"] Sep 11 15:31:22.180549 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:31:22.180517 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11322aa2_1920_420d_9d97_034446a0337d.slice/crio-b84e672a1050c34bd908d3e6e3373df8001140f4ec028ea54110687692ec0ddd WatchSource:0}: Error finding container b84e672a1050c34bd908d3e6e3373df8001140f4ec028ea54110687692ec0ddd: Status 404 returned error can't find the container with id b84e672a1050c34bd908d3e6e3373df8001140f4ec028ea54110687692ec0ddd Sep 11 15:31:22.182705 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:22.182669 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:31:22.182815 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:22.182732 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:31:22.182866 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:22.182831 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:31:22.534376 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:22.534285 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" event={"ID":"11322aa2-1920-420d-9d97-034446a0337d","Type":"ContainerStarted","Data":"7d7583a0c000e2a0ba114406dddc78ba9c0fb469c55d4962f23de770855a8b0c"} Sep 11 15:31:22.534376 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:22.534335 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" event={"ID":"11322aa2-1920-420d-9d97-034446a0337d","Type":"ContainerStarted","Data":"b84e672a1050c34bd908d3e6e3373df8001140f4ec028ea54110687692ec0ddd"} Sep 11 15:31:22.557819 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:22.557762 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" podStartSLOduration=1.5577280180000002 podStartE2EDuration="1.557728018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:31:22.55599396 +0000 UTC m=+1376.787918390" watchObservedRunningTime="2026-09-11 15:31:22.557728018 +0000 UTC m=+1376.789652447" Sep 11 15:31:22.892731 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:22.892700 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:23.723884 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:23.723848 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:23.897944 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:23.897912 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:24.520750 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:24.520703 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:24.544932 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:24.544906 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:24.545917 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:24.545896 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:24.619013 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:24.618981 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:25.307888 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:25.307856 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:26.312458 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:26.312423 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:27.389330 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:27.389291 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:28.083089 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.083056 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv"] Sep 11 15:31:28.083552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.083534 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e87a60f-0aa2-4c8c-97be-cefef2e0e84c" containerName="istio-proxy" Sep 11 15:31:28.087636 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.087614 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.090657 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.090551 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-808212-serving-cert\"" Sep 11 15:31:28.103488 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.103462 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv"] Sep 11 15:31:28.126223 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.126196 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:28.202198 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.202157 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-metrics-certs\") pod \"maas-api-e2e-api-a-808212-56f5dc59d-c78cv\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.202562 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.202535 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7z24\" (UniqueName: \"kubernetes.io/projected/98b1d749-fd43-4499-bfd8-ccd25140a5a7-kube-api-access-w7z24\") pod \"maas-api-e2e-api-a-808212-56f5dc59d-c78cv\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.202863 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.202843 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-maas-api-tls\") pod \"maas-api-e2e-api-a-808212-56f5dc59d-c78cv\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.203030 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.203013 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b1d749-fd43-4499-bfd8-ccd25140a5a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-808212-56f5dc59d-c78cv\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.227938 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.227907 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:28.304230 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.304204 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-maas-api-tls\") pod \"maas-api-e2e-api-a-808212-56f5dc59d-c78cv\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.304347 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.304236 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b1d749-fd43-4499-bfd8-ccd25140a5a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-808212-56f5dc59d-c78cv\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.304347 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.304305 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-metrics-certs\") pod \"maas-api-e2e-api-a-808212-56f5dc59d-c78cv\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.304483 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:31:28.304352 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-808212-serving-cert: secret "maas-api-e2e-api-a-808212-serving-cert" not found Sep 11 15:31:28.304483 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.304365 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7z24\" (UniqueName: \"kubernetes.io/projected/98b1d749-fd43-4499-bfd8-ccd25140a5a7-kube-api-access-w7z24\") pod \"maas-api-e2e-api-a-808212-56f5dc59d-c78cv\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.304483 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:31:28.304414 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-maas-api-tls podName:98b1d749-fd43-4499-bfd8-ccd25140a5a7 nodeName:}" failed. No retries permitted until 2026-09-11 15:31:28.804397211 +0000 UTC m=+1383.036321620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-maas-api-tls") pod "maas-api-e2e-api-a-808212-56f5dc59d-c78cv" (UID: "98b1d749-fd43-4499-bfd8-ccd25140a5a7") : secret "maas-api-e2e-api-a-808212-serving-cert" not found Sep 11 15:31:28.305027 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.305009 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b1d749-fd43-4499-bfd8-ccd25140a5a7-openshift-service-ca\") pod \"maas-api-e2e-api-a-808212-56f5dc59d-c78cv\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.306910 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.306891 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-metrics-certs\") pod \"maas-api-e2e-api-a-808212-56f5dc59d-c78cv\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.313377 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.313352 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7z24\" (UniqueName: \"kubernetes.io/projected/98b1d749-fd43-4499-bfd8-ccd25140a5a7-kube-api-access-w7z24\") pod \"maas-api-e2e-api-a-808212-56f5dc59d-c78cv\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.405360 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.405331 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:28.617893 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.617853 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:28.708773 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.708663 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:28.810941 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.810889 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-maas-api-tls\") pod \"maas-api-e2e-api-a-808212-56f5dc59d-c78cv\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.813486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.813462 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-maas-api-tls\") pod \"maas-api-e2e-api-a-808212-56f5dc59d-c78cv\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:28.998825 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:28.998720 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:29.271521 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:29.271492 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv"] Sep 11 15:31:29.272922 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:31:29.272897 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b1d749_fd43_4499_bfd8_ccd25140a5a7.slice/crio-d3cc37cdcd667d3b5a81d67f3e5a2063a73e4f985481c42179a2422e8a433aaf WatchSource:0}: Error finding container d3cc37cdcd667d3b5a81d67f3e5a2063a73e4f985481c42179a2422e8a433aaf: Status 404 returned error can't find the container with id d3cc37cdcd667d3b5a81d67f3e5a2063a73e4f985481c42179a2422e8a433aaf Sep 11 15:31:29.574298 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:29.574217 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" event={"ID":"98b1d749-fd43-4499-bfd8-ccd25140a5a7","Type":"ContainerStarted","Data":"d3cc37cdcd667d3b5a81d67f3e5a2063a73e4f985481c42179a2422e8a433aaf"} Sep 11 15:31:30.581821 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:30.581783 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" event={"ID":"98b1d749-fd43-4499-bfd8-ccd25140a5a7","Type":"ContainerStarted","Data":"09b7800395f1f6ddd15292826cdae7af783aa93991189bda348286da8e97ec58"} Sep 11 15:31:30.582311 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:30.581897 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:30.597690 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:30.597633 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" podStartSLOduration=2.102853873 podStartE2EDuration="2.597620128s" totalImagesPullingTime="494.766255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:31:30.596787982 +0000 UTC m=+1384.828712418" watchObservedRunningTime="2026-09-11 15:31:30.597620128 +0000 UTC m=+1384.829544558" Sep 11 15:31:35.282869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:35.282830 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:31:36.593096 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:36.593067 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:31:45.670434 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:45.670358 2558 generic.go:356] "Generic (PLEG): container finished" podID="11322aa2-1920-420d-9d97-034446a0337d" containerID="7d7583a0c000e2a0ba114406dddc78ba9c0fb469c55d4962f23de770855a8b0c" exitCode=0 Sep 11 15:31:45.670434 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:45.670402 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" event={"ID":"11322aa2-1920-420d-9d97-034446a0337d","Type":"ContainerDied","Data":"7d7583a0c000e2a0ba114406dddc78ba9c0fb469c55d4962f23de770855a8b0c"} Sep 11 15:31:45.670893 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:45.670831 2558 scope.go:117] "RemoveContainer" containerID="7d7583a0c000e2a0ba114406dddc78ba9c0fb469c55d4962f23de770855a8b0c" Sep 11 15:31:45.671988 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:45.671958 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:31:45.672071 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:45.672026 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:31:45.672071 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:45.672057 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:31:46.686588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:46.686539 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" event={"ID":"11322aa2-1920-420d-9d97-034446a0337d","Type":"ContainerStarted","Data":"76bd125be1ec63a73c6601958a81593b4bbd67ff199dba4c07c8bfe3d447244a"} Sep 11 15:31:46.893061 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:46.893018 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:46.893319 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:46.893294 2558 patch_prober.go:28] interesting pod/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 15:31:46.893417 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:46.893352 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" podUID="11322aa2-1920-420d-9d97-034446a0337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 15:31:47.893344 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:47.893306 2558 patch_prober.go:28] interesting pod/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 15:31:47.893773 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:47.893360 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" podUID="11322aa2-1920-420d-9d97-034446a0337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 15:31:48.892947 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:48.892909 2558 patch_prober.go:28] interesting pod/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 15:31:48.893111 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:48.892968 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" podUID="11322aa2-1920-420d-9d97-034446a0337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 15:31:49.892613 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:49.892578 2558 patch_prober.go:28] interesting pod/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 15:31:49.893043 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:49.892634 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" podUID="11322aa2-1920-420d-9d97-034446a0337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 15:31:50.417588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:50.417555 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:50.800184 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:50.800094 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:50.892921 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:50.892884 2558 patch_prober.go:28] interesting pod/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 15:31:50.893296 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:50.892956 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" podUID="11322aa2-1920-420d-9d97-034446a0337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 15:31:51.426852 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:51.426817 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:51.893355 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:51.893318 2558 patch_prober.go:28] interesting pod/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Sep 11 15:31:51.893844 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:51.893387 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" podUID="11322aa2-1920-420d-9d97-034446a0337d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Sep 11 15:31:52.106100 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:52.106065 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:52.385010 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:52.384921 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:31:52.767690 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:52.767653 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:52.897620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:52.897592 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:52.898013 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:52.897874 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:52.898641 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:52.898623 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:31:53.424145 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:53.424109 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:54.556437 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:54.556405 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:55.026921 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.026873 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:55.319672 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.319590 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr"] Sep 11 15:31:55.324352 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.324324 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.326965 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.326938 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-63c85a-openshift-default-dockercfg-txc5j\"" Sep 11 15:31:55.335774 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.335722 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr"] Sep 11 15:31:55.348068 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.348029 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:55.362346 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.362302 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-workload-certs\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.362543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.362366 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slxdr\" (UniqueName: \"kubernetes.io/projected/8037488e-698b-468a-a9ac-3c601d037e0a-kube-api-access-slxdr\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.362543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.362453 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8037488e-698b-468a-a9ac-3c601d037e0a-istiod-ca-cert\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.362543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.362505 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8037488e-698b-468a-a9ac-3c601d037e0a-istio-token\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.362543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.362528 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-workload-socket\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.362719 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.362553 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-istio-data\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.362719 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.362612 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-credential-socket\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.362719 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.362649 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-istio-envoy\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.362719 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.362683 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8037488e-698b-468a-a9ac-3c601d037e0a-istio-podinfo\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.463645 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.463599 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8037488e-698b-468a-a9ac-3c601d037e0a-istio-token\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.463869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.463657 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-workload-socket\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.463869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.463697 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-istio-data\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.463869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.463773 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-credential-socket\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.463869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.463811 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-istio-envoy\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.463869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.463846 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8037488e-698b-468a-a9ac-3c601d037e0a-istio-podinfo\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.464152 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.463893 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-workload-certs\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.464152 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.463933 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slxdr\" (UniqueName: \"kubernetes.io/projected/8037488e-698b-468a-a9ac-3c601d037e0a-kube-api-access-slxdr\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.464152 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.464006 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8037488e-698b-468a-a9ac-3c601d037e0a-istiod-ca-cert\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.464152 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.464135 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-istio-data\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.464367 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.464135 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-workload-socket\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.464367 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.464284 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-credential-socket\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.464620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.464598 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-workload-certs\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.464839 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.464681 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8037488e-698b-468a-a9ac-3c601d037e0a-istiod-ca-cert\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.466343 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.466315 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-istio-envoy\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.466683 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.466661 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8037488e-698b-468a-a9ac-3c601d037e0a-istio-podinfo\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.471678 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.471649 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8037488e-698b-468a-a9ac-3c601d037e0a-istio-token\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.472805 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.472758 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slxdr\" (UniqueName: \"kubernetes.io/projected/8037488e-698b-468a-a9ac-3c601d037e0a-kube-api-access-slxdr\") pod \"e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.638489 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.638391 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:55.797694 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.797652 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:55.919700 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.919672 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr"] Sep 11 15:31:55.921141 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:31:55.921114 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8037488e_698b_468a_a9ac_3c601d037e0a.slice/crio-82b22329ebef3717ad3af50e90fef063511fde650ba259ae88001585bf8109bc WatchSource:0}: Error finding container 82b22329ebef3717ad3af50e90fef063511fde650ba259ae88001585bf8109bc: Status 404 returned error can't find the container with id 82b22329ebef3717ad3af50e90fef063511fde650ba259ae88001585bf8109bc Sep 11 15:31:55.923921 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.923887 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:31:55.923991 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.923958 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:31:55.924033 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:55.923993 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:31:56.749648 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:56.749613 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" event={"ID":"8037488e-698b-468a-a9ac-3c601d037e0a","Type":"ContainerStarted","Data":"7b46e53114c08c8d059a28c206389968a4eb451004cf693184a544842ea223e8"} Sep 11 15:31:56.750049 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:56.749676 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" event={"ID":"8037488e-698b-468a-a9ac-3c601d037e0a","Type":"ContainerStarted","Data":"82b22329ebef3717ad3af50e90fef063511fde650ba259ae88001585bf8109bc"} Sep 11 15:31:56.772849 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:56.772794 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" podStartSLOduration=1.7727771940000001 podStartE2EDuration="1.772777194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:31:56.77069705 +0000 UTC m=+1411.002621486" watchObservedRunningTime="2026-09-11 15:31:56.772777194 +0000 UTC m=+1411.004701624" Sep 11 15:31:57.638640 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:57.638596 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:57.643812 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:57.643784 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:57.755724 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:57.755693 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:57.756851 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:57.756833 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:31:58.085073 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:58.085039 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:58.674720 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:58.674683 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:58.775088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:58.775054 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:31:59.635621 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:31:59.635587 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:00.503835 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:00.503807 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:00.604091 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:00.604037 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:01.236180 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:01.236151 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:01.921559 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:01.921529 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:02.696027 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:02.695995 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:02.797173 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:02.797140 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:03.105908 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:03.105798 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:03.206088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:03.206056 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:03.278906 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:03.278872 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:09.192750 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:09.192677 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:10.197636 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:10.197596 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:12.585570 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:12.585528 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:32:17.526973 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:17.526932 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:21.275207 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:21.275172 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:22.917228 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:22.917193 2558 generic.go:356] "Generic (PLEG): container finished" podID="8037488e-698b-468a-a9ac-3c601d037e0a" containerID="7b46e53114c08c8d059a28c206389968a4eb451004cf693184a544842ea223e8" exitCode=0 Sep 11 15:32:22.917680 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:22.917272 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" event={"ID":"8037488e-698b-468a-a9ac-3c601d037e0a","Type":"ContainerDied","Data":"7b46e53114c08c8d059a28c206389968a4eb451004cf693184a544842ea223e8"} Sep 11 15:32:22.917723 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:22.917707 2558 scope.go:117] "RemoveContainer" containerID="7b46e53114c08c8d059a28c206389968a4eb451004cf693184a544842ea223e8" Sep 11 15:32:22.918902 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:22.918873 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:32:22.918990 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:22.918937 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:32:22.918990 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:22.918975 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:32:23.594661 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:23.594625 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:23.925104 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:23.925074 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" event={"ID":"8037488e-698b-468a-a9ac-3c601d037e0a","Type":"ContainerStarted","Data":"cffc9b1faf87c33e7e903eb087fbabccfceb8513690b0d6e54dbaf1121cf8271"} Sep 11 15:32:24.638963 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:24.638924 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:32:24.640263 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:24.640233 2558 patch_prober.go:28] interesting pod/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 15:32:24.640403 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:24.640285 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" podUID="8037488e-698b-468a-a9ac-3c601d037e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 15:32:25.029569 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:25.029539 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:25.639477 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:25.639439 2558 patch_prober.go:28] interesting pod/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 15:32:25.639666 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:25.639494 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" podUID="8037488e-698b-468a-a9ac-3c601d037e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 15:32:25.831210 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:25.831177 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:26.639717 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:26.639676 2558 patch_prober.go:28] interesting pod/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 15:32:26.640133 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:26.639766 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" podUID="8037488e-698b-468a-a9ac-3c601d037e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 15:32:26.797253 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:26.797221 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:27.510958 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:27.510916 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:27.639815 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:27.639774 2558 patch_prober.go:28] interesting pod/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 15:32:27.640201 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:27.639844 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" podUID="8037488e-698b-468a-a9ac-3c601d037e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 15:32:27.756098 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:27.756046 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:32:28.204824 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:28.204789 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:28.638811 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:28.638708 2558 patch_prober.go:28] interesting pod/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 15:32:28.638811 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:28.638790 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" podUID="8037488e-698b-468a-a9ac-3c601d037e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 15:32:28.781384 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:28.781347 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:28.893695 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:28.893662 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:29.643688 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:29.643655 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:32:29.644769 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:29.644733 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:32:29.736477 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:29.732494 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:30.983940 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:30.983903 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:32:31.834258 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:31.834217 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:32.624423 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:32.624389 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:33.505653 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:33.505614 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:34.248502 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:34.248433 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:34.950536 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:34.950502 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:35.733482 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:35.733425 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:36.428423 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:36.428392 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:37.192700 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:37.192667 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:39.730464 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:39.730431 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:39.784836 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:39.784808 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:41.505765 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:41.505720 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:47.263534 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.263500 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv"] Sep 11 15:32:47.264031 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.263768 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" podUID="98b1d749-fd43-4499-bfd8-ccd25140a5a7" containerName="maas-api" containerID="cri-o://09b7800395f1f6ddd15292826cdae7af783aa93991189bda348286da8e97ec58" gracePeriod=30 Sep 11 15:32:47.289616 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.289588 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:47.356196 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.356154 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:47.456453 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.456413 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:47.505718 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.505694 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:32:47.512377 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.512352 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:47.573331 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.573248 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b1d749-fd43-4499-bfd8-ccd25140a5a7-openshift-service-ca\") pod \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " Sep 11 15:32:47.573331 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.573313 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-metrics-certs\") pod \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " Sep 11 15:32:47.573578 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.573340 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-maas-api-tls\") pod \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " Sep 11 15:32:47.573578 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.573362 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7z24\" (UniqueName: \"kubernetes.io/projected/98b1d749-fd43-4499-bfd8-ccd25140a5a7-kube-api-access-w7z24\") pod \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\" (UID: \"98b1d749-fd43-4499-bfd8-ccd25140a5a7\") " Sep 11 15:32:47.573703 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.573590 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b1d749-fd43-4499-bfd8-ccd25140a5a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "98b1d749-fd43-4499-bfd8-ccd25140a5a7" (UID: "98b1d749-fd43-4499-bfd8-ccd25140a5a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:32:47.575554 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.575525 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "98b1d749-fd43-4499-bfd8-ccd25140a5a7" (UID: "98b1d749-fd43-4499-bfd8-ccd25140a5a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:32:47.575554 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.575537 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98b1d749-fd43-4499-bfd8-ccd25140a5a7-kube-api-access-w7z24" (OuterVolumeSpecName: "kube-api-access-w7z24") pod "98b1d749-fd43-4499-bfd8-ccd25140a5a7" (UID: "98b1d749-fd43-4499-bfd8-ccd25140a5a7"). InnerVolumeSpecName "kube-api-access-w7z24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:32:47.575712 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.575616 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "98b1d749-fd43-4499-bfd8-ccd25140a5a7" (UID: "98b1d749-fd43-4499-bfd8-ccd25140a5a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:32:47.674808 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.674767 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98b1d749-fd43-4499-bfd8-ccd25140a5a7-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:32:47.674808 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.674799 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-metrics-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:32:47.674808 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.674810 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98b1d749-fd43-4499-bfd8-ccd25140a5a7-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:32:47.674808 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.674818 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7z24\" (UniqueName: \"kubernetes.io/projected/98b1d749-fd43-4499-bfd8-ccd25140a5a7-kube-api-access-w7z24\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:32:47.833639 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.833559 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:47.933337 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:47.933309 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:48.067178 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:48.067148 2558 generic.go:356] "Generic (PLEG): container finished" podID="98b1d749-fd43-4499-bfd8-ccd25140a5a7" containerID="09b7800395f1f6ddd15292826cdae7af783aa93991189bda348286da8e97ec58" exitCode=0 Sep 11 15:32:48.067383 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:48.067213 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" Sep 11 15:32:48.067383 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:48.067242 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" event={"ID":"98b1d749-fd43-4499-bfd8-ccd25140a5a7","Type":"ContainerDied","Data":"09b7800395f1f6ddd15292826cdae7af783aa93991189bda348286da8e97ec58"} Sep 11 15:32:48.067383 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:48.067300 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv" event={"ID":"98b1d749-fd43-4499-bfd8-ccd25140a5a7","Type":"ContainerDied","Data":"d3cc37cdcd667d3b5a81d67f3e5a2063a73e4f985481c42179a2422e8a433aaf"} Sep 11 15:32:48.067383 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:48.067324 2558 scope.go:117] "RemoveContainer" containerID="09b7800395f1f6ddd15292826cdae7af783aa93991189bda348286da8e97ec58" Sep 11 15:32:48.077500 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:48.077483 2558 scope.go:117] "RemoveContainer" containerID="09b7800395f1f6ddd15292826cdae7af783aa93991189bda348286da8e97ec58" Sep 11 15:32:48.077772 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:32:48.077750 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b7800395f1f6ddd15292826cdae7af783aa93991189bda348286da8e97ec58\": container with ID starting with 09b7800395f1f6ddd15292826cdae7af783aa93991189bda348286da8e97ec58 not found: ID does not exist" containerID="09b7800395f1f6ddd15292826cdae7af783aa93991189bda348286da8e97ec58" Sep 11 15:32:48.077873 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:48.077778 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b7800395f1f6ddd15292826cdae7af783aa93991189bda348286da8e97ec58"} err="failed to get container status \"09b7800395f1f6ddd15292826cdae7af783aa93991189bda348286da8e97ec58\": rpc error: code = NotFound desc = could not find container \"09b7800395f1f6ddd15292826cdae7af783aa93991189bda348286da8e97ec58\": container with ID starting with 09b7800395f1f6ddd15292826cdae7af783aa93991189bda348286da8e97ec58 not found: ID does not exist" Sep 11 15:32:48.089506 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:48.089446 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv"] Sep 11 15:32:48.094597 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:48.094577 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-808212-56f5dc59d-c78cv"] Sep 11 15:32:48.265097 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:48.265068 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98b1d749-fd43-4499-bfd8-ccd25140a5a7" path="/var/lib/kubelet/pods/98b1d749-fd43-4499-bfd8-ccd25140a5a7/volumes" Sep 11 15:32:48.613766 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:48.613725 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:48.796556 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:48.796518 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:32:49.696080 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:49.696035 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:50.424381 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:50.424340 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:51.116174 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:51.116142 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:51.884829 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:51.884799 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:52.653326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:52.653290 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:53.636551 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:53.636521 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:53.736811 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:53.736774 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:53.882291 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:53.882254 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs"] Sep 11 15:32:53.882520 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:53.882482 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" podUID="11322aa2-1920-420d-9d97-034446a0337d" containerName="istio-proxy" containerID="cri-o://76bd125be1ec63a73c6601958a81593b4bbd67ff199dba4c07c8bfe3d447244a" gracePeriod=30 Sep 11 15:32:53.903065 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:53.903041 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:54.347261 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:54.347179 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:55.191578 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:55.191542 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:56.023235 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:56.023203 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:56.676585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:56.676552 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:57.312692 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:57.312660 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:58.017344 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:58.017311 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:58.718834 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:58.718804 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:32:59.133375 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.133346 2558 generic.go:356] "Generic (PLEG): container finished" podID="11322aa2-1920-420d-9d97-034446a0337d" containerID="76bd125be1ec63a73c6601958a81593b4bbd67ff199dba4c07c8bfe3d447244a" exitCode=0 Sep 11 15:32:59.133691 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.133384 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" event={"ID":"11322aa2-1920-420d-9d97-034446a0337d","Type":"ContainerDied","Data":"76bd125be1ec63a73c6601958a81593b4bbd67ff199dba4c07c8bfe3d447244a"} Sep 11 15:32:59.133691 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.133454 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" event={"ID":"11322aa2-1920-420d-9d97-034446a0337d","Type":"ContainerDied","Data":"b84e672a1050c34bd908d3e6e3373df8001140f4ec028ea54110687692ec0ddd"} Sep 11 15:32:59.133691 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.133474 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b84e672a1050c34bd908d3e6e3373df8001140f4ec028ea54110687692ec0ddd" Sep 11 15:32:59.133691 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.133487 2558 scope.go:117] "RemoveContainer" containerID="7d7583a0c000e2a0ba114406dddc78ba9c0fb469c55d4962f23de770855a8b0c" Sep 11 15:32:59.138770 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.138733 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:32:59.292689 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.292601 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11322aa2-1920-420d-9d97-034446a0337d-istio-token\") pod \"11322aa2-1920-420d-9d97-034446a0337d\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " Sep 11 15:32:59.292689 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.292649 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lppp\" (UniqueName: \"kubernetes.io/projected/11322aa2-1920-420d-9d97-034446a0337d-kube-api-access-5lppp\") pod \"11322aa2-1920-420d-9d97-034446a0337d\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " Sep 11 15:32:59.292955 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.292694 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-istio-data\") pod \"11322aa2-1920-420d-9d97-034446a0337d\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " Sep 11 15:32:59.292955 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.292736 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-istio-envoy\") pod \"11322aa2-1920-420d-9d97-034446a0337d\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " Sep 11 15:32:59.292955 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.292809 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-workload-socket\") pod \"11322aa2-1920-420d-9d97-034446a0337d\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " Sep 11 15:32:59.292955 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.292831 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-credential-socket\") pod \"11322aa2-1920-420d-9d97-034446a0337d\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " Sep 11 15:32:59.292955 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.292893 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11322aa2-1920-420d-9d97-034446a0337d-istiod-ca-cert\") pod \"11322aa2-1920-420d-9d97-034446a0337d\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " Sep 11 15:32:59.292955 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.292930 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11322aa2-1920-420d-9d97-034446a0337d-istio-podinfo\") pod \"11322aa2-1920-420d-9d97-034446a0337d\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " Sep 11 15:32:59.293284 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.293010 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-istio-data" (OuterVolumeSpecName: "istio-data") pod "11322aa2-1920-420d-9d97-034446a0337d" (UID: "11322aa2-1920-420d-9d97-034446a0337d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:32:59.293284 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.293084 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11322aa2-1920-420d-9d97-034446a0337d" (UID: "11322aa2-1920-420d-9d97-034446a0337d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:32:59.293284 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.293125 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11322aa2-1920-420d-9d97-034446a0337d" (UID: "11322aa2-1920-420d-9d97-034446a0337d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:32:59.293444 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.293360 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11322aa2-1920-420d-9d97-034446a0337d" (UID: "11322aa2-1920-420d-9d97-034446a0337d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:32:59.293444 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.293387 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11322aa2-1920-420d-9d97-034446a0337d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11322aa2-1920-420d-9d97-034446a0337d" (UID: "11322aa2-1920-420d-9d97-034446a0337d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:32:59.293444 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.293433 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-workload-certs\") pod \"11322aa2-1920-420d-9d97-034446a0337d\" (UID: \"11322aa2-1920-420d-9d97-034446a0337d\") " Sep 11 15:32:59.293728 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.293711 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:32:59.293829 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.293734 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:32:59.293829 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.293765 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11322aa2-1920-420d-9d97-034446a0337d-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:32:59.293829 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.293777 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:32:59.293829 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.293788 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:32:59.294895 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.294872 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11322aa2-1920-420d-9d97-034446a0337d-kube-api-access-5lppp" (OuterVolumeSpecName: "kube-api-access-5lppp") pod "11322aa2-1920-420d-9d97-034446a0337d" (UID: "11322aa2-1920-420d-9d97-034446a0337d"). InnerVolumeSpecName "kube-api-access-5lppp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:32:59.295006 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.294954 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11322aa2-1920-420d-9d97-034446a0337d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11322aa2-1920-420d-9d97-034446a0337d" (UID: "11322aa2-1920-420d-9d97-034446a0337d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:32:59.295279 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.295264 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11322aa2-1920-420d-9d97-034446a0337d" (UID: "11322aa2-1920-420d-9d97-034446a0337d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:32:59.295349 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.295280 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11322aa2-1920-420d-9d97-034446a0337d-istio-token" (OuterVolumeSpecName: "istio-token") pod "11322aa2-1920-420d-9d97-034446a0337d" (UID: "11322aa2-1920-420d-9d97-034446a0337d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:32:59.395000 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.394977 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11322aa2-1920-420d-9d97-034446a0337d-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:32:59.395000 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.394999 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11322aa2-1920-420d-9d97-034446a0337d-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:32:59.395205 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.395010 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lppp\" (UniqueName: \"kubernetes.io/projected/11322aa2-1920-420d-9d97-034446a0337d-kube-api-access-5lppp\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:32:59.395205 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:32:59.395019 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11322aa2-1920-420d-9d97-034446a0337d-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:00.140920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:00.140893 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs" Sep 11 15:33:00.172715 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:00.172677 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs"] Sep 11 15:33:00.176193 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:00.176165 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-808212-openshift-default-66845f8df4-mj7bs"] Sep 11 15:33:00.264906 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:00.264880 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11322aa2-1920-420d-9d97-034446a0337d" path="/var/lib/kubelet/pods/11322aa2-1920-420d-9d97-034446a0337d/volumes" Sep 11 15:33:07.080099 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:07.080068 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:33:09.736941 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:09.736899 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:09.805089 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:09.805047 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:09.866902 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:09.866872 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:09.968560 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:09.968527 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:10.292271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:10.292236 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:10.369442 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:10.369409 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:11.145188 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:11.145149 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:11.787582 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:11.787541 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:12.694949 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:12.694920 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:13.363867 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:13.363839 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:15.943577 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:15.943540 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:16.065370 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:16.065334 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:16.190129 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:16.190091 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr"] Sep 11 15:33:16.190364 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:16.190341 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" podUID="8037488e-698b-468a-a9ac-3c601d037e0a" containerName="istio-proxy" containerID="cri-o://cffc9b1faf87c33e7e903eb087fbabccfceb8513690b0d6e54dbaf1121cf8271" gracePeriod=30 Sep 11 15:33:16.203287 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:16.203224 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:16.809131 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:16.809100 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:17.620580 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:17.620548 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:18.386804 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:18.386767 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:19.029054 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:19.029022 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:19.829590 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:19.829559 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:20.472539 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:20.472470 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:21.447524 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.447499 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:33:21.500548 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.500460 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8037488e-698b-468a-a9ac-3c601d037e0a-istio-podinfo\") pod \"8037488e-698b-468a-a9ac-3c601d037e0a\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " Sep 11 15:33:21.500548 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.500503 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slxdr\" (UniqueName: \"kubernetes.io/projected/8037488e-698b-468a-a9ac-3c601d037e0a-kube-api-access-slxdr\") pod \"8037488e-698b-468a-a9ac-3c601d037e0a\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " Sep 11 15:33:21.500548 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.500524 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-credential-socket\") pod \"8037488e-698b-468a-a9ac-3c601d037e0a\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " Sep 11 15:33:21.501147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.500562 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-workload-socket\") pod \"8037488e-698b-468a-a9ac-3c601d037e0a\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " Sep 11 15:33:21.501147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.500592 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8037488e-698b-468a-a9ac-3c601d037e0a-istio-token\") pod \"8037488e-698b-468a-a9ac-3c601d037e0a\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " Sep 11 15:33:21.501147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.500633 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-istio-data\") pod \"8037488e-698b-468a-a9ac-3c601d037e0a\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " Sep 11 15:33:21.501147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.500652 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-workload-certs\") pod \"8037488e-698b-468a-a9ac-3c601d037e0a\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " Sep 11 15:33:21.501147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.500949 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8037488e-698b-468a-a9ac-3c601d037e0a" (UID: "8037488e-698b-468a-a9ac-3c601d037e0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:21.501147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.500963 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8037488e-698b-468a-a9ac-3c601d037e0a" (UID: "8037488e-698b-468a-a9ac-3c601d037e0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:21.501147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.500955 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-istio-envoy\") pod \"8037488e-698b-468a-a9ac-3c601d037e0a\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " Sep 11 15:33:21.501147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.500976 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-istio-data" (OuterVolumeSpecName: "istio-data") pod "8037488e-698b-468a-a9ac-3c601d037e0a" (UID: "8037488e-698b-468a-a9ac-3c601d037e0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:21.501147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.501043 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8037488e-698b-468a-a9ac-3c601d037e0a-istiod-ca-cert\") pod \"8037488e-698b-468a-a9ac-3c601d037e0a\" (UID: \"8037488e-698b-468a-a9ac-3c601d037e0a\") " Sep 11 15:33:21.501620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.501164 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8037488e-698b-468a-a9ac-3c601d037e0a" (UID: "8037488e-698b-468a-a9ac-3c601d037e0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:21.501620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.501381 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8037488e-698b-468a-a9ac-3c601d037e0a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8037488e-698b-468a-a9ac-3c601d037e0a" (UID: "8037488e-698b-468a-a9ac-3c601d037e0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:33:21.501620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.501476 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8037488e-698b-468a-a9ac-3c601d037e0a-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:21.501620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.501496 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:21.501620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.501512 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:21.501620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.501525 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:21.501620 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.501537 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:21.502924 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.502892 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8037488e-698b-468a-a9ac-3c601d037e0a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8037488e-698b-468a-a9ac-3c601d037e0a" (UID: "8037488e-698b-468a-a9ac-3c601d037e0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:33:21.503050 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.502964 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8037488e-698b-468a-a9ac-3c601d037e0a-istio-token" (OuterVolumeSpecName: "istio-token") pod "8037488e-698b-468a-a9ac-3c601d037e0a" (UID: "8037488e-698b-468a-a9ac-3c601d037e0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:33:21.503050 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.502993 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8037488e-698b-468a-a9ac-3c601d037e0a" (UID: "8037488e-698b-468a-a9ac-3c601d037e0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:21.503050 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.503000 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8037488e-698b-468a-a9ac-3c601d037e0a-kube-api-access-slxdr" (OuterVolumeSpecName: "kube-api-access-slxdr") pod "8037488e-698b-468a-a9ac-3c601d037e0a" (UID: "8037488e-698b-468a-a9ac-3c601d037e0a"). InnerVolumeSpecName "kube-api-access-slxdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:33:21.602174 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.602140 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8037488e-698b-468a-a9ac-3c601d037e0a-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:21.602174 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.602166 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8037488e-698b-468a-a9ac-3c601d037e0a-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:21.602174 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.602177 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slxdr\" (UniqueName: \"kubernetes.io/projected/8037488e-698b-468a-a9ac-3c601d037e0a-kube-api-access-slxdr\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:21.602174 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:21.602186 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8037488e-698b-468a-a9ac-3c601d037e0a-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:22.266992 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:22.266956 2558 generic.go:356] "Generic (PLEG): container finished" podID="8037488e-698b-468a-a9ac-3c601d037e0a" containerID="cffc9b1faf87c33e7e903eb087fbabccfceb8513690b0d6e54dbaf1121cf8271" exitCode=0 Sep 11 15:33:22.267146 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:22.267002 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" Sep 11 15:33:22.267146 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:22.267035 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" event={"ID":"8037488e-698b-468a-a9ac-3c601d037e0a","Type":"ContainerDied","Data":"cffc9b1faf87c33e7e903eb087fbabccfceb8513690b0d6e54dbaf1121cf8271"} Sep 11 15:33:22.267146 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:22.267089 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr" event={"ID":"8037488e-698b-468a-a9ac-3c601d037e0a","Type":"ContainerDied","Data":"82b22329ebef3717ad3af50e90fef063511fde650ba259ae88001585bf8109bc"} Sep 11 15:33:22.267146 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:22.267112 2558 scope.go:117] "RemoveContainer" containerID="cffc9b1faf87c33e7e903eb087fbabccfceb8513690b0d6e54dbaf1121cf8271" Sep 11 15:33:22.277241 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:22.277217 2558 scope.go:117] "RemoveContainer" containerID="7b46e53114c08c8d059a28c206389968a4eb451004cf693184a544842ea223e8" Sep 11 15:33:22.285598 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:22.285569 2558 scope.go:117] "RemoveContainer" containerID="cffc9b1faf87c33e7e903eb087fbabccfceb8513690b0d6e54dbaf1121cf8271" Sep 11 15:33:22.286213 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:33:22.286159 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cffc9b1faf87c33e7e903eb087fbabccfceb8513690b0d6e54dbaf1121cf8271\": container with ID starting with cffc9b1faf87c33e7e903eb087fbabccfceb8513690b0d6e54dbaf1121cf8271 not found: ID does not exist" containerID="cffc9b1faf87c33e7e903eb087fbabccfceb8513690b0d6e54dbaf1121cf8271" Sep 11 15:33:22.286213 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:22.286200 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cffc9b1faf87c33e7e903eb087fbabccfceb8513690b0d6e54dbaf1121cf8271"} err="failed to get container status \"cffc9b1faf87c33e7e903eb087fbabccfceb8513690b0d6e54dbaf1121cf8271\": rpc error: code = NotFound desc = could not find container \"cffc9b1faf87c33e7e903eb087fbabccfceb8513690b0d6e54dbaf1121cf8271\": container with ID starting with cffc9b1faf87c33e7e903eb087fbabccfceb8513690b0d6e54dbaf1121cf8271 not found: ID does not exist" Sep 11 15:33:22.286385 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:22.286225 2558 scope.go:117] "RemoveContainer" containerID="7b46e53114c08c8d059a28c206389968a4eb451004cf693184a544842ea223e8" Sep 11 15:33:22.286664 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:33:22.286635 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b46e53114c08c8d059a28c206389968a4eb451004cf693184a544842ea223e8\": container with ID starting with 7b46e53114c08c8d059a28c206389968a4eb451004cf693184a544842ea223e8 not found: ID does not exist" containerID="7b46e53114c08c8d059a28c206389968a4eb451004cf693184a544842ea223e8" Sep 11 15:33:22.286783 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:22.286671 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b46e53114c08c8d059a28c206389968a4eb451004cf693184a544842ea223e8"} err="failed to get container status \"7b46e53114c08c8d059a28c206389968a4eb451004cf693184a544842ea223e8\": rpc error: code = NotFound desc = could not find container \"7b46e53114c08c8d059a28c206389968a4eb451004cf693184a544842ea223e8\": container with ID starting with 7b46e53114c08c8d059a28c206389968a4eb451004cf693184a544842ea223e8 not found: ID does not exist" Sep 11 15:33:22.289072 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:22.289052 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr"] Sep 11 15:33:22.292543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:22.292520 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-63c85a-openshift-default-55cd559c46-5wqpr"] Sep 11 15:33:22.982255 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:22.982215 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:33:24.270846 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:24.270802 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8037488e-698b-468a-a9ac-3c601d037e0a" path="/var/lib/kubelet/pods/8037488e-698b-468a-a9ac-3c601d037e0a/volumes" Sep 11 15:33:26.553492 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:26.553457 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:27.272897 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:27.272863 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:27.367406 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:27.367374 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:27.444309 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:27.444273 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:27.506150 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:27.506103 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:27.783541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:27.783506 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:27.884404 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:27.884373 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:28.661549 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:28.661512 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:28.757327 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:28.757246 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:29.101332 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.101252 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:29.365162 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.365065 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst"] Sep 11 15:33:29.365633 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.365616 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8037488e-698b-468a-a9ac-3c601d037e0a" containerName="istio-proxy" Sep 11 15:33:29.365732 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.365635 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8037488e-698b-468a-a9ac-3c601d037e0a" containerName="istio-proxy" Sep 11 15:33:29.365732 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.365648 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11322aa2-1920-420d-9d97-034446a0337d" containerName="istio-proxy" Sep 11 15:33:29.365732 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.365657 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="11322aa2-1920-420d-9d97-034446a0337d" containerName="istio-proxy" Sep 11 15:33:29.365732 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.365680 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b1d749-fd43-4499-bfd8-ccd25140a5a7" containerName="maas-api" Sep 11 15:33:29.365732 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.365688 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b1d749-fd43-4499-bfd8-ccd25140a5a7" containerName="maas-api" Sep 11 15:33:29.366002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.365795 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="11322aa2-1920-420d-9d97-034446a0337d" containerName="istio-proxy" Sep 11 15:33:29.366002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.365810 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="98b1d749-fd43-4499-bfd8-ccd25140a5a7" containerName="maas-api" Sep 11 15:33:29.366002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.365821 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8037488e-698b-468a-a9ac-3c601d037e0a" containerName="istio-proxy" Sep 11 15:33:29.366002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.365830 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8037488e-698b-468a-a9ac-3c601d037e0a" containerName="istio-proxy" Sep 11 15:33:29.369364 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.369341 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.370159 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.370130 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:29.371865 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.371843 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8559ee-openshift-default-dockercfg-tgwf5\"" Sep 11 15:33:29.380214 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.380195 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst"] Sep 11 15:33:29.468711 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.468669 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:29.472904 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.472874 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-workload-socket\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.473047 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.472960 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-workload-certs\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.473047 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.472985 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-podinfo\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.473047 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.473002 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-envoy\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.473047 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.473020 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-data\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.473235 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.473084 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-token\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.473235 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.473159 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-credential-socket\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.473235 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.473190 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e0eaca-1572-4be3-bb22-a0b471e06941-istiod-ca-cert\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.473235 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.473218 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkfnx\" (UniqueName: \"kubernetes.io/projected/93e0eaca-1572-4be3-bb22-a0b471e06941-kube-api-access-dkfnx\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.569982 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.569919 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:29.573920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.573890 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-workload-certs\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.573920 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.573922 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-podinfo\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.574127 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.573938 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-envoy\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.574127 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.573956 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-data\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.574127 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.573989 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-token\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.574127 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.574021 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-credential-socket\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.574350 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.574247 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e0eaca-1572-4be3-bb22-a0b471e06941-istiod-ca-cert\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.574350 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.574312 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkfnx\" (UniqueName: \"kubernetes.io/projected/93e0eaca-1572-4be3-bb22-a0b471e06941-kube-api-access-dkfnx\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.574456 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.574344 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-workload-certs\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.574456 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.574396 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-data\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.574456 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.574444 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-workload-socket\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.574622 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.574579 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-credential-socket\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.574792 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.574757 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-workload-socket\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.575012 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.574991 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e0eaca-1572-4be3-bb22-a0b471e06941-istiod-ca-cert\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.576494 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.576475 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-envoy\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.576752 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.576716 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-podinfo\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.582067 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.582046 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkfnx\" (UniqueName: \"kubernetes.io/projected/93e0eaca-1572-4be3-bb22-a0b471e06941-kube-api-access-dkfnx\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.582329 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.582310 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-token\") pod \"e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.681134 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.681108 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:29.956509 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.956480 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst"] Sep 11 15:33:29.957833 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:33:29.957807 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e0eaca_1572_4be3_bb22_a0b471e06941.slice/crio-37bee9ed6c8520d9d26d842a5b1ba818e98d513a0d07d9ab44a4cb795ae61622 WatchSource:0}: Error finding container 37bee9ed6c8520d9d26d842a5b1ba818e98d513a0d07d9ab44a4cb795ae61622: Status 404 returned error can't find the container with id 37bee9ed6c8520d9d26d842a5b1ba818e98d513a0d07d9ab44a4cb795ae61622 Sep 11 15:33:29.959948 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.959914 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:33:29.960059 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.959993 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:33:29.960059 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:29.960036 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:33:30.311693 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:30.311590 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:30.322121 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:30.322094 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" event={"ID":"93e0eaca-1572-4be3-bb22-a0b471e06941","Type":"ContainerStarted","Data":"c40535ef65103bf53d9f7c4bb421879fc7ea5d58db11d407b5b733d6e0fea353"} Sep 11 15:33:30.322279 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:30.322140 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" event={"ID":"93e0eaca-1572-4be3-bb22-a0b471e06941","Type":"ContainerStarted","Data":"37bee9ed6c8520d9d26d842a5b1ba818e98d513a0d07d9ab44a4cb795ae61622"} Sep 11 15:33:30.343751 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:30.343695 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" podStartSLOduration=1.343682043 podStartE2EDuration="1.343682043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:33:30.340652118 +0000 UTC m=+1504.572576554" watchObservedRunningTime="2026-09-11 15:33:30.343682043 +0000 UTC m=+1504.575606474" Sep 11 15:33:30.681897 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:30.681865 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:31.687512 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:31.687479 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:32.165266 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:32.165223 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:32.265259 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:32.265220 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:32.336675 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:32.336648 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:32.337959 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:32.337924 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:32.369226 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:32.369193 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:32.915604 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:32.915572 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:33.811774 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:33.811731 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:33.943883 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:33.943851 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:34.043829 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:34.043791 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:34.251941 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:34.251911 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2"] Sep 11 15:33:34.252146 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:34.252123 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" podUID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerName="istio-proxy" containerID="cri-o://702ff745be88bb54f918f397dda079ea5f8c05a896f5359333dcbf1975777179" gracePeriod=30 Sep 11 15:33:34.272888 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:34.272855 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:34.617536 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:34.617435 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:35.411843 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:35.411805 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:35.513559 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:35.513525 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:35.620292 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:35.620257 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:35.973519 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:35.973480 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:36.071421 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:36.071383 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:36.329716 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:36.329634 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:37.231996 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:37.231969 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:37.909424 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:37.909396 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:38.611393 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:38.611361 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:38.690712 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:38.690678 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:33:39.398498 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.398459 2558 generic.go:356] "Generic (PLEG): container finished" podID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerID="702ff745be88bb54f918f397dda079ea5f8c05a896f5359333dcbf1975777179" exitCode=0 Sep 11 15:33:39.398649 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.398530 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" event={"ID":"b78f8075-db10-4642-8d1f-cd7850ede4ef","Type":"ContainerDied","Data":"702ff745be88bb54f918f397dda079ea5f8c05a896f5359333dcbf1975777179"} Sep 11 15:33:39.398649 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.398599 2558 scope.go:117] "RemoveContainer" containerID="b9ae2bc09f49ce20444f297287697c99bdbab9fa84df59044f96e1220f8b6532" Sep 11 15:33:39.434314 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.434290 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:39.512541 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.512520 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:33:39.570828 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.570652 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-credential-socket\") pod \"b78f8075-db10-4642-8d1f-cd7850ede4ef\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " Sep 11 15:33:39.570828 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.570791 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-envoy\") pod \"b78f8075-db10-4642-8d1f-cd7850ede4ef\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " Sep 11 15:33:39.571053 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.570830 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-workload-certs\") pod \"b78f8075-db10-4642-8d1f-cd7850ede4ef\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " Sep 11 15:33:39.571053 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.570856 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78f8075-db10-4642-8d1f-cd7850ede4ef-istiod-ca-cert\") pod \"b78f8075-db10-4642-8d1f-cd7850ede4ef\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " Sep 11 15:33:39.571053 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.570896 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkgmr\" (UniqueName: \"kubernetes.io/projected/b78f8075-db10-4642-8d1f-cd7850ede4ef-kube-api-access-tkgmr\") pod \"b78f8075-db10-4642-8d1f-cd7850ede4ef\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " Sep 11 15:33:39.571053 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.570929 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-workload-socket\") pod \"b78f8075-db10-4642-8d1f-cd7850ede4ef\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " Sep 11 15:33:39.571053 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.570962 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-podinfo\") pod \"b78f8075-db10-4642-8d1f-cd7850ede4ef\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " Sep 11 15:33:39.571053 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.571012 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-data\") pod \"b78f8075-db10-4642-8d1f-cd7850ede4ef\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " Sep 11 15:33:39.571053 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.571048 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-token\") pod \"b78f8075-db10-4642-8d1f-cd7850ede4ef\" (UID: \"b78f8075-db10-4642-8d1f-cd7850ede4ef\") " Sep 11 15:33:39.571687 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.571660 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b78f8075-db10-4642-8d1f-cd7850ede4ef" (UID: "b78f8075-db10-4642-8d1f-cd7850ede4ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:39.571793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.571678 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b78f8075-db10-4642-8d1f-cd7850ede4ef" (UID: "b78f8075-db10-4642-8d1f-cd7850ede4ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:39.572084 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.572060 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b78f8075-db10-4642-8d1f-cd7850ede4ef" (UID: "b78f8075-db10-4642-8d1f-cd7850ede4ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:39.572275 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.572072 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "b78f8075-db10-4642-8d1f-cd7850ede4ef" (UID: "b78f8075-db10-4642-8d1f-cd7850ede4ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:39.572485 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.572456 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78f8075-db10-4642-8d1f-cd7850ede4ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b78f8075-db10-4642-8d1f-cd7850ede4ef" (UID: "b78f8075-db10-4642-8d1f-cd7850ede4ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:33:39.574238 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.574211 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "b78f8075-db10-4642-8d1f-cd7850ede4ef" (UID: "b78f8075-db10-4642-8d1f-cd7850ede4ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:33:39.575068 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.575042 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b78f8075-db10-4642-8d1f-cd7850ede4ef" (UID: "b78f8075-db10-4642-8d1f-cd7850ede4ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:33:39.575516 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.575491 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b78f8075-db10-4642-8d1f-cd7850ede4ef" (UID: "b78f8075-db10-4642-8d1f-cd7850ede4ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:33:39.575631 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.575570 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78f8075-db10-4642-8d1f-cd7850ede4ef-kube-api-access-tkgmr" (OuterVolumeSpecName: "kube-api-access-tkgmr") pod "b78f8075-db10-4642-8d1f-cd7850ede4ef" (UID: "b78f8075-db10-4642-8d1f-cd7850ede4ef"). InnerVolumeSpecName "kube-api-access-tkgmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:33:39.672853 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.672818 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:39.672853 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.672845 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:39.672853 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.672855 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:39.672853 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.672864 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:39.673326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.672873 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:39.673326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.672880 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78f8075-db10-4642-8d1f-cd7850ede4ef-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:39.673326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.672888 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkgmr\" (UniqueName: \"kubernetes.io/projected/b78f8075-db10-4642-8d1f-cd7850ede4ef-kube-api-access-tkgmr\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:39.673326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.672897 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78f8075-db10-4642-8d1f-cd7850ede4ef-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:39.673326 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:39.672907 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78f8075-db10-4642-8d1f-cd7850ede4ef-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:40.406172 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:40.406142 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" Sep 11 15:33:40.406172 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:40.406160 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2" event={"ID":"b78f8075-db10-4642-8d1f-cd7850ede4ef","Type":"ContainerDied","Data":"dc3d2e92d8be1a6476ca481cfe920b8bf01ed4333f77c9659a45047238f0a5bc"} Sep 11 15:33:40.406383 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:40.406210 2558 scope.go:117] "RemoveContainer" containerID="702ff745be88bb54f918f397dda079ea5f8c05a896f5359333dcbf1975777179" Sep 11 15:33:40.425484 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:40.425458 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2"] Sep 11 15:33:40.429613 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:40.429588 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4a348b-openshift-default-67fdc8f898-z4nn2"] Sep 11 15:33:42.269335 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:42.269289 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b78f8075-db10-4642-8d1f-cd7850ede4ef" path="/var/lib/kubelet/pods/b78f8075-db10-4642-8d1f-cd7850ede4ef/volumes" Sep 11 15:33:44.704305 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:44.704253 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:45.234886 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.234854 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq"] Sep 11 15:33:45.235350 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.235319 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" podUID="99bb113b-83ad-44c2-ab2d-3e35a5990220" containerName="maas-api" containerID="cri-o://4f30ef15bf830720118eefdd6464c3c224d693e4ada46dd1e974f809edc1001c" gracePeriod=30 Sep 11 15:33:45.287770 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.287714 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:45.427423 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.427394 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:45.437083 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.437057 2558 generic.go:356] "Generic (PLEG): container finished" podID="99bb113b-83ad-44c2-ab2d-3e35a5990220" containerID="4f30ef15bf830720118eefdd6464c3c224d693e4ada46dd1e974f809edc1001c" exitCode=0 Sep 11 15:33:45.437255 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.437127 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" event={"ID":"99bb113b-83ad-44c2-ab2d-3e35a5990220","Type":"ContainerDied","Data":"4f30ef15bf830720118eefdd6464c3c224d693e4ada46dd1e974f809edc1001c"} Sep 11 15:33:45.531585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.531548 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:45.535902 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.535882 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:33:45.628136 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.628105 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q9pd\" (UniqueName: \"kubernetes.io/projected/99bb113b-83ad-44c2-ab2d-3e35a5990220-kube-api-access-7q9pd\") pod \"99bb113b-83ad-44c2-ab2d-3e35a5990220\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " Sep 11 15:33:45.628327 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.628145 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bb113b-83ad-44c2-ab2d-3e35a5990220-maas-api-tls\") pod \"99bb113b-83ad-44c2-ab2d-3e35a5990220\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " Sep 11 15:33:45.628327 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.628190 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bb113b-83ad-44c2-ab2d-3e35a5990220-openshift-service-ca\") pod \"99bb113b-83ad-44c2-ab2d-3e35a5990220\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " Sep 11 15:33:45.628327 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.628229 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:45.628327 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.628272 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bb113b-83ad-44c2-ab2d-3e35a5990220-metrics-certs\") pod \"99bb113b-83ad-44c2-ab2d-3e35a5990220\" (UID: \"99bb113b-83ad-44c2-ab2d-3e35a5990220\") " Sep 11 15:33:45.628722 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.628624 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99bb113b-83ad-44c2-ab2d-3e35a5990220-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "99bb113b-83ad-44c2-ab2d-3e35a5990220" (UID: "99bb113b-83ad-44c2-ab2d-3e35a5990220"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:33:45.630526 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.630503 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99bb113b-83ad-44c2-ab2d-3e35a5990220-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "99bb113b-83ad-44c2-ab2d-3e35a5990220" (UID: "99bb113b-83ad-44c2-ab2d-3e35a5990220"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:33:45.630844 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.630824 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99bb113b-83ad-44c2-ab2d-3e35a5990220-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "99bb113b-83ad-44c2-ab2d-3e35a5990220" (UID: "99bb113b-83ad-44c2-ab2d-3e35a5990220"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:33:45.630907 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.630828 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bb113b-83ad-44c2-ab2d-3e35a5990220-kube-api-access-7q9pd" (OuterVolumeSpecName: "kube-api-access-7q9pd") pod "99bb113b-83ad-44c2-ab2d-3e35a5990220" (UID: "99bb113b-83ad-44c2-ab2d-3e35a5990220"). InnerVolumeSpecName "kube-api-access-7q9pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:33:45.729216 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.729183 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99bb113b-83ad-44c2-ab2d-3e35a5990220-metrics-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:45.729216 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.729213 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q9pd\" (UniqueName: \"kubernetes.io/projected/99bb113b-83ad-44c2-ab2d-3e35a5990220-kube-api-access-7q9pd\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:45.729216 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.729225 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99bb113b-83ad-44c2-ab2d-3e35a5990220-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:45.729644 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.729234 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bb113b-83ad-44c2-ab2d-3e35a5990220-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:33:45.900299 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:45.900269 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:46.004953 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:46.004921 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:46.444663 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:46.444637 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" Sep 11 15:33:46.444663 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:46.444653 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq" event={"ID":"99bb113b-83ad-44c2-ab2d-3e35a5990220","Type":"ContainerDied","Data":"1f130f2b992ef9785ecf46bd7c823676a47f05a9550aa948576788e568c397c7"} Sep 11 15:33:46.444897 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:46.444718 2558 scope.go:117] "RemoveContainer" containerID="4f30ef15bf830720118eefdd6464c3c224d693e4ada46dd1e974f809edc1001c" Sep 11 15:33:46.464685 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:46.464651 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq"] Sep 11 15:33:46.469346 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:46.469318 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7ee57f-84d7dbd58b-qw4tq"] Sep 11 15:33:46.727935 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:46.727911 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:47.406839 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:47.406789 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:48.265458 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:48.265429 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99bb113b-83ad-44c2-ab2d-3e35a5990220" path="/var/lib/kubelet/pods/99bb113b-83ad-44c2-ab2d-3e35a5990220/volumes" Sep 11 15:33:48.323117 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:48.323074 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:48.377898 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:48.377864 2558 patch_prober.go:28] interesting pod/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 15:33:48.378096 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:48.377930 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 15:33:49.204589 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:49.204481 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:49.883994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:49.883862 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:50.470795 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:50.470733 2558 generic.go:356] "Generic (PLEG): container finished" podID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerID="c40535ef65103bf53d9f7c4bb421879fc7ea5d58db11d407b5b733d6e0fea353" exitCode=0 Sep 11 15:33:50.471269 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:50.470814 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" event={"ID":"93e0eaca-1572-4be3-bb22-a0b471e06941","Type":"ContainerDied","Data":"c40535ef65103bf53d9f7c4bb421879fc7ea5d58db11d407b5b733d6e0fea353"} Sep 11 15:33:50.471336 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:50.471311 2558 scope.go:117] "RemoveContainer" containerID="c40535ef65103bf53d9f7c4bb421879fc7ea5d58db11d407b5b733d6e0fea353" Sep 11 15:33:50.472400 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:50.472371 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:33:50.472494 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:50.472435 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:33:50.472494 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:50.472465 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:33:50.639942 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:50.639905 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:51.182010 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:51.181971 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:33:51.413782 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:51.413750 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:51.482437 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:51.482330 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" event={"ID":"93e0eaca-1572-4be3-bb22-a0b471e06941","Type":"ContainerStarted","Data":"66b7b7bb1ac8b3c708afddb1328b34ee7c49e53c11cee318b73f75c1e7d14c39"} Sep 11 15:33:51.681830 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:51.681797 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:51.682449 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:51.682337 2558 patch_prober.go:28] interesting pod/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 11 15:33:51.682671 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:51.682646 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 11 15:33:52.041337 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:52.041302 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:52.142961 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:52.142928 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:52.268354 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:52.268310 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:52.430987 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:52.430953 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:52.682511 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:52.682418 2558 patch_prober.go:28] interesting pod/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 11 15:33:52.682511 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:52.682475 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 11 15:33:52.738576 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:52.738545 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:53.030306 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.030221 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf"] Sep 11 15:33:53.030834 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.030814 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99bb113b-83ad-44c2-ab2d-3e35a5990220" containerName="maas-api" Sep 11 15:33:53.030889 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.030841 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="99bb113b-83ad-44c2-ab2d-3e35a5990220" containerName="maas-api" Sep 11 15:33:53.030889 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.030857 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerName="istio-proxy" Sep 11 15:33:53.030889 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.030865 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerName="istio-proxy" Sep 11 15:33:53.030984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.030918 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerName="istio-proxy" Sep 11 15:33:53.030984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.030927 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerName="istio-proxy" Sep 11 15:33:53.030984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.030946 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8037488e-698b-468a-a9ac-3c601d037e0a" containerName="istio-proxy" Sep 11 15:33:53.030984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.030954 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8037488e-698b-468a-a9ac-3c601d037e0a" containerName="istio-proxy" Sep 11 15:33:53.030984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.030965 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11322aa2-1920-420d-9d97-034446a0337d" containerName="istio-proxy" Sep 11 15:33:53.030984 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.030973 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="11322aa2-1920-420d-9d97-034446a0337d" containerName="istio-proxy" Sep 11 15:33:53.031160 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.031070 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerName="istio-proxy" Sep 11 15:33:53.031160 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.031084 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="11322aa2-1920-420d-9d97-034446a0337d" containerName="istio-proxy" Sep 11 15:33:53.031160 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.031096 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b78f8075-db10-4642-8d1f-cd7850ede4ef" containerName="istio-proxy" Sep 11 15:33:53.031160 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.031106 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="99bb113b-83ad-44c2-ab2d-3e35a5990220" containerName="maas-api" Sep 11 15:33:53.034787 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.034767 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.037008 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.036984 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0ce6e9-openshift-default-dockercfg-tgphq\"" Sep 11 15:33:53.044166 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.044141 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf"] Sep 11 15:33:53.069315 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.069267 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:53.101316 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.101278 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-envoy\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.101497 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.101325 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-credential-socket\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.101497 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.101360 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-workload-socket\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.101497 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.101417 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-podinfo\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.101497 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.101451 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istiod-ca-cert\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.101497 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.101478 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-workload-certs\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.101765 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.101537 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-data\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.101765 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.101563 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-token\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.101765 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.101621 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9xqw\" (UniqueName: \"kubernetes.io/projected/bfd22d1f-1195-4f3c-a007-542819ae4d1f-kube-api-access-t9xqw\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.202325 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.202278 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-podinfo\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.202517 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.202332 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istiod-ca-cert\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.202517 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.202363 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-workload-certs\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.202517 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.202409 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-data\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.202517 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.202434 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-token\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.202517 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.202499 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9xqw\" (UniqueName: \"kubernetes.io/projected/bfd22d1f-1195-4f3c-a007-542819ae4d1f-kube-api-access-t9xqw\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.202854 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.202529 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-envoy\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.202854 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.202566 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-credential-socket\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.202854 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.202600 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-workload-socket\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.203280 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.203164 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-data\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.203280 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.203191 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-workload-certs\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.203280 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.203247 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-credential-socket\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.203488 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.203430 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istiod-ca-cert\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.203488 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.203437 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-workload-socket\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.205274 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.205254 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-envoy\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.205487 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.205470 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-podinfo\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.210095 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.210075 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-token\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.210291 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.210274 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9xqw\" (UniqueName: \"kubernetes.io/projected/bfd22d1f-1195-4f3c-a007-542819ae4d1f-kube-api-access-t9xqw\") pod \"e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.346857 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.346733 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:53.646720 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.646692 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf"] Sep 11 15:33:53.648326 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:33:53.648299 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd22d1f_1195_4f3c_a007_542819ae4d1f.slice/crio-463bd4a1e94de97b16a734524290e269fe56fe4a315991be7312d2342281cc1f WatchSource:0}: Error finding container 463bd4a1e94de97b16a734524290e269fe56fe4a315991be7312d2342281cc1f: Status 404 returned error can't find the container with id 463bd4a1e94de97b16a734524290e269fe56fe4a315991be7312d2342281cc1f Sep 11 15:33:53.650865 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.650832 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:33:53.650982 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.650906 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:33:53.650982 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.650956 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:33:53.681983 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.681549 2558 patch_prober.go:28] interesting pod/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 11 15:33:53.681983 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.681612 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 11 15:33:53.766970 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:53.766933 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:54.500615 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:54.500576 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" event={"ID":"bfd22d1f-1195-4f3c-a007-542819ae4d1f","Type":"ContainerStarted","Data":"9d71dd9dc62afec67f90a3361d7463e8806b92479ed2d88a64fe0721b26db395"} Sep 11 15:33:54.500865 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:54.500671 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" event={"ID":"bfd22d1f-1195-4f3c-a007-542819ae4d1f","Type":"ContainerStarted","Data":"463bd4a1e94de97b16a734524290e269fe56fe4a315991be7312d2342281cc1f"} Sep 11 15:33:54.527387 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:54.527338 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" podStartSLOduration=1.5273219519999999 podStartE2EDuration="1.527321952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:33:54.524431127 +0000 UTC m=+1528.756355588" watchObservedRunningTime="2026-09-11 15:33:54.527321952 +0000 UTC m=+1528.759246383" Sep 11 15:33:54.682084 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:54.682049 2558 patch_prober.go:28] interesting pod/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 11 15:33:54.682244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:54.682104 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 11 15:33:55.347408 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:55.347368 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:55.358451 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:55.358423 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:55.506005 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:55.505972 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:55.507109 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:55.507085 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:33:55.682276 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:55.682240 2558 patch_prober.go:28] interesting pod/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 11 15:33:55.682496 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:55.682309 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 11 15:33:56.020446 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:56.020332 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:56.688891 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:56.688856 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:56.689459 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:56.689430 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:56.690360 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:56.690335 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:33:56.717155 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:56.717125 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:56.813427 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:56.813401 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:56.960465 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:56.960382 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:57.818760 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:57.818708 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:58.652072 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:58.652027 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:59.318450 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:59.318419 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:59.419571 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:59.419532 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:59.669656 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:59.669625 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:59.779533 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:59.779501 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:33:59.877098 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:33:59.877067 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:34:00.182126 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:00.182080 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:34:00.912573 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:00.912541 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:34:04.184132 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:04.184101 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:34:15.626553 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:15.626520 2558 generic.go:356] "Generic (PLEG): container finished" podID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerID="9d71dd9dc62afec67f90a3361d7463e8806b92479ed2d88a64fe0721b26db395" exitCode=0 Sep 11 15:34:15.627002 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:15.626594 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" event={"ID":"bfd22d1f-1195-4f3c-a007-542819ae4d1f","Type":"ContainerDied","Data":"9d71dd9dc62afec67f90a3361d7463e8806b92479ed2d88a64fe0721b26db395"} Sep 11 15:34:15.627049 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:15.627030 2558 scope.go:117] "RemoveContainer" containerID="9d71dd9dc62afec67f90a3361d7463e8806b92479ed2d88a64fe0721b26db395" Sep 11 15:34:15.628218 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:15.628192 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:34:15.628279 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:15.628257 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:34:15.628316 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:15.628293 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 15:34:16.634820 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:16.634781 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" event={"ID":"bfd22d1f-1195-4f3c-a007-542819ae4d1f","Type":"ContainerStarted","Data":"6b04937e63dc1d932c4b05de9160bb7128ab414a247b2393643853aa30dd1477"} Sep 11 15:34:17.347438 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:17.347408 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:34:17.347662 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:17.347639 2558 patch_prober.go:28] interesting pod/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 15:34:17.347713 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:17.347695 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" podUID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 15:34:18.347371 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:18.347333 2558 patch_prober.go:28] interesting pod/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 15:34:18.347799 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:18.347402 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" podUID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 15:34:19.347703 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:19.347666 2558 patch_prober.go:28] interesting pod/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 15:34:19.348086 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:19.347720 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" podUID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 15:34:20.347242 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:20.347205 2558 patch_prober.go:28] interesting pod/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 15:34:20.347416 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:20.347257 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" podUID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 15:34:21.312676 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:21.312641 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:34:21.348052 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:21.348028 2558 patch_prober.go:28] interesting pod/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 15:34:21.348145 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:21.348074 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" podUID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 15:34:21.763948 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:21.763911 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:34:22.047704 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:22.047587 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:34:22.351811 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:22.351720 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:34:22.352258 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:22.351999 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:34:22.352770 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:22.352752 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:34:22.821415 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:22.821382 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:34:25.815019 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:25.814988 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:34:32.972416 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:32.972369 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl"] Sep 11 15:34:32.976451 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:32.976434 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:32.979832 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:32.979804 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8559ee\"/\"ipp-route-8559ee-kserve-self-signed-certs\"" Sep 11 15:34:32.979983 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:32.979958 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8559ee\"/\"openshift-service-ca.crt\"" Sep 11 15:34:32.980062 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:32.979984 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8559ee\"/\"default-dockercfg-cxqn4\"" Sep 11 15:34:32.980062 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:32.979810 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8559ee\"/\"kube-root-ca.crt\"" Sep 11 15:34:32.988487 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:32.988285 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl"] Sep 11 15:34:33.000214 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.000190 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:34:33.092768 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.092711 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-tmp-dir\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.092768 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.092762 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-tls-certs\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.092978 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.092793 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-dshm\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.092978 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.092876 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm6vt\" (UniqueName: \"kubernetes.io/projected/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-kube-api-access-wm6vt\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.092978 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.092938 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-home\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.092978 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.092974 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-model-cache\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.194338 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.194304 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-home\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.194485 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.194347 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-model-cache\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.194485 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.194389 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-tmp-dir\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.194485 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.194403 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-tls-certs\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.194485 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.194419 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-dshm\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.194485 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.194451 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm6vt\" (UniqueName: \"kubernetes.io/projected/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-kube-api-access-wm6vt\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.194798 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.194783 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-home\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.194875 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.194820 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-tmp-dir\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.194875 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.194839 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-model-cache\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.196652 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.196632 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-dshm\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.196994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.196979 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-tls-certs\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.202059 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.202039 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm6vt\" (UniqueName: \"kubernetes.io/projected/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-kube-api-access-wm6vt\") pod \"ipp-route-8559ee-kserve-67cb5d65c9-jlhdl\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.291409 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.291350 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:33.562419 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.562340 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl"] Sep 11 15:34:33.565542 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:34:33.565519 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c3bc3c9_f277_4423_bed9_3e3f666c0fac.slice/crio-31c075fac8ec8440f4d5835a042a25a33d44e3ad6ecd1c3b84d0c06a26447658 WatchSource:0}: Error finding container 31c075fac8ec8440f4d5835a042a25a33d44e3ad6ecd1c3b84d0c06a26447658: Status 404 returned error can't find the container with id 31c075fac8ec8440f4d5835a042a25a33d44e3ad6ecd1c3b84d0c06a26447658 Sep 11 15:34:33.735823 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.735784 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" event={"ID":"0c3bc3c9-f277-4423-bed9-3e3f666c0fac","Type":"ContainerStarted","Data":"a362ef507ecb050e3206c6542acb7367bbe33a3882ca9583f093704146059bb9"} Sep 11 15:34:33.735976 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.735833 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" event={"ID":"0c3bc3c9-f277-4423-bed9-3e3f666c0fac","Type":"ContainerStarted","Data":"31c075fac8ec8440f4d5835a042a25a33d44e3ad6ecd1c3b84d0c06a26447658"} Sep 11 15:34:33.751732 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:33.751683 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" podStartSLOduration=1.7516672309999999 podStartE2EDuration="1.751667231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:34:33.750583117 +0000 UTC m=+1567.982507551" watchObservedRunningTime="2026-09-11 15:34:33.751667231 +0000 UTC m=+1567.983591662" Sep 11 15:34:37.282869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:37.282834 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:34:43.292194 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:43.292158 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:43.292644 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:43.292407 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:43.305052 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:43.305023 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:43.812662 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:43.812636 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:34:52.876949 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:52.876910 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:34:55.121511 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:55.121461 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:34:55.176722 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:55.176692 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:34:59.782518 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:34:59.782475 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:00.075085 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:00.075005 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:00.988451 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:00.988418 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:02.277274 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:02.277233 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:02.849891 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:02.849854 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:02.940432 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:02.940399 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:03.813866 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:03.813833 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:04.571562 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:04.571527 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:05.240425 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.240384 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:05.903525 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.903494 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc"] Sep 11 15:35:05.907339 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.907311 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:05.909343 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.909321 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 15:35:05.910433 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.910280 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:35:05.910433 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.910324 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 15:35:05.910433 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.910332 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-f58c81-serving-cert\"" Sep 11 15:35:05.910433 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.910324 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-79rjx\"" Sep 11 15:35:05.915952 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.915927 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc"] Sep 11 15:35:05.941941 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.941907 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:05.995104 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.995074 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b518458-3317-473a-aaa1-dfea1b0896d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:05.995259 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.995132 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:05.995259 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.995192 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:05.995259 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.995232 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwfpg\" (UniqueName: \"kubernetes.io/projected/3b518458-3317-473a-aaa1-dfea1b0896d6-kube-api-access-dwfpg\") pod \"maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:05.996796 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:05.996780 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:06.096371 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:06.096333 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b518458-3317-473a-aaa1-dfea1b0896d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:06.096619 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:06.096394 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:06.096619 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:35:06.096486 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-serving-cert: secret "maas-api-e2e-ipp-cleanup-f58c81-serving-cert" not found Sep 11 15:35:06.096619 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:06.096501 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:06.096619 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:35:06.096543 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-maas-api-tls podName:3b518458-3317-473a-aaa1-dfea1b0896d6 nodeName:}" failed. No retries permitted until 2026-09-11 15:35:06.596527744 +0000 UTC m=+1600.828452164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" (UID: "3b518458-3317-473a-aaa1-dfea1b0896d6") : secret "maas-api-e2e-ipp-cleanup-f58c81-serving-cert" not found Sep 11 15:35:06.096619 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:06.096564 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwfpg\" (UniqueName: \"kubernetes.io/projected/3b518458-3317-473a-aaa1-dfea1b0896d6-kube-api-access-dwfpg\") pod \"maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:06.097019 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:06.096999 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b518458-3317-473a-aaa1-dfea1b0896d6-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:06.098896 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:06.098878 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:06.107470 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:06.107446 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwfpg\" (UniqueName: \"kubernetes.io/projected/3b518458-3317-473a-aaa1-dfea1b0896d6-kube-api-access-dwfpg\") pod \"maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:06.394235 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:06.394202 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:06.487287 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:06.487231 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:06.601489 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:06.601421 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:06.604560 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:06.604532 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:06.823027 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:06.822931 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:07.097695 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:07.094969 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc"] Sep 11 15:35:07.098289 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:35:07.098260 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b518458_3317_473a_aaa1_dfea1b0896d6.slice/crio-2f7b05feb4ccd2b1a02ed009f8124cd48ef1a03372a2d3fe559fd56e0905588c WatchSource:0}: Error finding container 2f7b05feb4ccd2b1a02ed009f8124cd48ef1a03372a2d3fe559fd56e0905588c: Status 404 returned error can't find the container with id 2f7b05feb4ccd2b1a02ed009f8124cd48ef1a03372a2d3fe559fd56e0905588c Sep 11 15:35:07.099674 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:07.099656 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 15:35:07.936016 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:07.935982 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" event={"ID":"3b518458-3317-473a-aaa1-dfea1b0896d6","Type":"ContainerStarted","Data":"99118973e765fb5bb91d23061b266dc730af083865028a4e5f9e3c0fd6ae80a8"} Sep 11 15:35:07.936387 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:07.936034 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" event={"ID":"3b518458-3317-473a-aaa1-dfea1b0896d6","Type":"ContainerStarted","Data":"2f7b05feb4ccd2b1a02ed009f8124cd48ef1a03372a2d3fe559fd56e0905588c"} Sep 11 15:35:07.936387 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:07.936186 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:07.952711 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:07.952667 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" podStartSLOduration=2.509998235 podStartE2EDuration="2.952651979s" totalImagesPullingTime="442.653744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:35:07.950186297 +0000 UTC m=+1602.182110739" watchObservedRunningTime="2026-09-11 15:35:07.952651979 +0000 UTC m=+1602.184576409" Sep 11 15:35:10.684338 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:10.684298 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:35:13.947577 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:13.947545 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:28.984658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:28.984621 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:29.613470 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:29.613438 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:30.352569 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:30.352529 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:31.015021 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:31.014983 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:31.807268 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:31.807234 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:32.444234 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:32.444205 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:32.943939 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:32.943908 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:34.520447 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:34.520412 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:35.148847 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:35.148810 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:35.801600 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:35.801570 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:36.532453 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:36.532419 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:37.158848 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:37.158818 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:37.896680 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:37.896649 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:44.598188 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.598152 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc"] Sep 11 15:35:44.598628 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.598403 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" podUID="3b518458-3317-473a-aaa1-dfea1b0896d6" containerName="maas-api" containerID="cri-o://99118973e765fb5bb91d23061b266dc730af083865028a4e5f9e3c0fd6ae80a8" gracePeriod=30 Sep 11 15:35:44.658335 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.658305 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:44.793597 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.793563 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:44.853089 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.853030 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:44.858607 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.858584 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:44.945519 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.945491 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:44.966657 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.966633 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b518458-3317-473a-aaa1-dfea1b0896d6-openshift-service-ca\") pod \"3b518458-3317-473a-aaa1-dfea1b0896d6\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " Sep 11 15:35:44.966791 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.966773 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-metrics-certs\") pod \"3b518458-3317-473a-aaa1-dfea1b0896d6\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " Sep 11 15:35:44.966846 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.966819 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwfpg\" (UniqueName: \"kubernetes.io/projected/3b518458-3317-473a-aaa1-dfea1b0896d6-kube-api-access-dwfpg\") pod \"3b518458-3317-473a-aaa1-dfea1b0896d6\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " Sep 11 15:35:44.966909 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.966846 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-maas-api-tls\") pod \"3b518458-3317-473a-aaa1-dfea1b0896d6\" (UID: \"3b518458-3317-473a-aaa1-dfea1b0896d6\") " Sep 11 15:35:44.967069 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.967046 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b518458-3317-473a-aaa1-dfea1b0896d6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b518458-3317-473a-aaa1-dfea1b0896d6" (UID: "3b518458-3317-473a-aaa1-dfea1b0896d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:35:44.967196 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.967181 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b518458-3317-473a-aaa1-dfea1b0896d6-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:35:44.968905 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.968882 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3b518458-3317-473a-aaa1-dfea1b0896d6" (UID: "3b518458-3317-473a-aaa1-dfea1b0896d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:35:44.968990 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.968898 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b518458-3317-473a-aaa1-dfea1b0896d6" (UID: "3b518458-3317-473a-aaa1-dfea1b0896d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:35:44.968990 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:44.968917 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b518458-3317-473a-aaa1-dfea1b0896d6-kube-api-access-dwfpg" (OuterVolumeSpecName: "kube-api-access-dwfpg") pod "3b518458-3317-473a-aaa1-dfea1b0896d6" (UID: "3b518458-3317-473a-aaa1-dfea1b0896d6"). InnerVolumeSpecName "kube-api-access-dwfpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:35:45.006172 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.006149 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:45.068502 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.068471 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-metrics-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:35:45.068502 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.068498 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwfpg\" (UniqueName: \"kubernetes.io/projected/3b518458-3317-473a-aaa1-dfea1b0896d6-kube-api-access-dwfpg\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:35:45.068699 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.068513 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b518458-3317-473a-aaa1-dfea1b0896d6-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:35:45.150588 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.150558 2558 generic.go:356] "Generic (PLEG): container finished" podID="3b518458-3317-473a-aaa1-dfea1b0896d6" containerID="99118973e765fb5bb91d23061b266dc730af083865028a4e5f9e3c0fd6ae80a8" exitCode=0 Sep 11 15:35:45.150813 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.150607 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" Sep 11 15:35:45.150813 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.150652 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" event={"ID":"3b518458-3317-473a-aaa1-dfea1b0896d6","Type":"ContainerDied","Data":"99118973e765fb5bb91d23061b266dc730af083865028a4e5f9e3c0fd6ae80a8"} Sep 11 15:35:45.150813 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.150712 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc" event={"ID":"3b518458-3317-473a-aaa1-dfea1b0896d6","Type":"ContainerDied","Data":"2f7b05feb4ccd2b1a02ed009f8124cd48ef1a03372a2d3fe559fd56e0905588c"} Sep 11 15:35:45.150813 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.150777 2558 scope.go:117] "RemoveContainer" containerID="99118973e765fb5bb91d23061b266dc730af083865028a4e5f9e3c0fd6ae80a8" Sep 11 15:35:45.160486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.160467 2558 scope.go:117] "RemoveContainer" containerID="99118973e765fb5bb91d23061b266dc730af083865028a4e5f9e3c0fd6ae80a8" Sep 11 15:35:45.160795 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:35:45.160771 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99118973e765fb5bb91d23061b266dc730af083865028a4e5f9e3c0fd6ae80a8\": container with ID starting with 99118973e765fb5bb91d23061b266dc730af083865028a4e5f9e3c0fd6ae80a8 not found: ID does not exist" containerID="99118973e765fb5bb91d23061b266dc730af083865028a4e5f9e3c0fd6ae80a8" Sep 11 15:35:45.160881 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.160802 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99118973e765fb5bb91d23061b266dc730af083865028a4e5f9e3c0fd6ae80a8"} err="failed to get container status \"99118973e765fb5bb91d23061b266dc730af083865028a4e5f9e3c0fd6ae80a8\": rpc error: code = NotFound desc = could not find container \"99118973e765fb5bb91d23061b266dc730af083865028a4e5f9e3c0fd6ae80a8\": container with ID starting with 99118973e765fb5bb91d23061b266dc730af083865028a4e5f9e3c0fd6ae80a8 not found: ID does not exist" Sep 11 15:35:45.171705 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.171682 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc"] Sep 11 15:35:45.177268 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.177237 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f58c81-5b8bf446c6-prgqc"] Sep 11 15:35:45.324913 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.324880 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:45.426263 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:45.426190 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:46.069929 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:46.069900 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:46.270416 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:46.270380 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b518458-3317-473a-aaa1-dfea1b0896d6" path="/var/lib/kubelet/pods/3b518458-3317-473a-aaa1-dfea1b0896d6/volumes" Sep 11 15:35:46.916904 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:46.916870 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:47.564212 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:47.564180 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:48.194365 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:48.194336 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:48.983488 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:48.983458 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:35:51.010543 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:51.010492 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:51.074195 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:51.074165 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:51.283357 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:51.283273 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:55.012773 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.012724 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf"] Sep 11 15:35:55.013287 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.013268 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b518458-3317-473a-aaa1-dfea1b0896d6" containerName="maas-api" Sep 11 15:35:55.013362 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.013291 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b518458-3317-473a-aaa1-dfea1b0896d6" containerName="maas-api" Sep 11 15:35:55.013418 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.013392 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b518458-3317-473a-aaa1-dfea1b0896d6" containerName="maas-api" Sep 11 15:35:55.017560 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.017539 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf" Sep 11 15:35:55.020517 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.020485 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fzvxj\"" Sep 11 15:35:55.020517 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.020506 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:35:55.023714 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.023689 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf"] Sep 11 15:35:55.126393 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.126355 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:55.161130 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.161104 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf\" (UID: \"e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf" Sep 11 15:35:55.261603 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.261575 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf\" (UID: \"e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf" Sep 11 15:35:55.262171 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.262151 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf\" (UID: \"e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf" Sep 11 15:35:55.329802 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.329720 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf" Sep 11 15:35:55.599798 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:55.599773 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf"] Sep 11 15:35:55.601799 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:35:55.601771 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80a24a3_f83e_4e57_8a0c_a8a384b2c7bb.slice/crio-dcd5645350caf019749c9536a7673928b7d2fd55270f10377a7f545b19427508 WatchSource:0}: Error finding container dcd5645350caf019749c9536a7673928b7d2fd55270f10377a7f545b19427508: Status 404 returned error can't find the container with id dcd5645350caf019749c9536a7673928b7d2fd55270f10377a7f545b19427508 Sep 11 15:35:56.001191 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:56.001143 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:56.215648 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:56.215610 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf" event={"ID":"e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb","Type":"ContainerStarted","Data":"4c8e3962b0b4503e74f591ec09ec530c58732bc70e246ecc5f2cda8f759d9926"} Sep 11 15:35:56.216072 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:56.215665 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf" event={"ID":"e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb","Type":"ContainerStarted","Data":"dcd5645350caf019749c9536a7673928b7d2fd55270f10377a7f545b19427508"} Sep 11 15:35:56.232459 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:56.232414 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf" podStartSLOduration=2.232399255 podStartE2EDuration="2.232399255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:35:56.230752251 +0000 UTC m=+1650.462676682" watchObservedRunningTime="2026-09-11 15:35:56.232399255 +0000 UTC m=+1650.464323684" Sep 11 15:35:56.921409 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:56.921373 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:57.222636 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:57.222537 2558 generic.go:356] "Generic (PLEG): container finished" podID="e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb" containerID="4c8e3962b0b4503e74f591ec09ec530c58732bc70e246ecc5f2cda8f759d9926" exitCode=0 Sep 11 15:35:57.222636 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:57.222623 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf" event={"ID":"e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb","Type":"ContainerDied","Data":"4c8e3962b0b4503e74f591ec09ec530c58732bc70e246ecc5f2cda8f759d9926"} Sep 11 15:35:57.618642 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:57.618545 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:58.362402 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:58.362377 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf" Sep 11 15:35:58.409434 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:58.409405 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:58.493261 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:58.493178 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb-maas-api-service-ca\") pod \"e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb\" (UID: \"e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb\") " Sep 11 15:35:58.493564 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:58.493540 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb" (UID: "e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:35:58.594715 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:58.594686 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb-maas-api-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:35:59.076526 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:59.076496 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:35:59.235928 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:59.235891 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf" event={"ID":"e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb","Type":"ContainerDied","Data":"dcd5645350caf019749c9536a7673928b7d2fd55270f10377a7f545b19427508"} Sep 11 15:35:59.235928 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:59.235927 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf" Sep 11 15:35:59.236174 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:35:59.235940 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcd5645350caf019749c9536a7673928b7d2fd55270f10377a7f545b19427508" Sep 11 15:36:08.185694 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:08.185659 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:36:10.607639 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:10.607608 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:10.672246 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:10.672216 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:10.737706 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:10.737675 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:10.807207 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:10.807175 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf"] Sep 11 15:36:10.808426 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:10.808406 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:10.809618 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:10.809600 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8559ee-l8tvf"] Sep 11 15:36:10.869180 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:10.869106 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:11.201368 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:11.201331 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:11.301715 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:11.301685 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:11.916423 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:11.916393 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:12.264454 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:12.264377 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb" path="/var/lib/kubelet/pods/e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb/volumes" Sep 11 15:36:12.662333 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:12.662306 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:13.420111 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:13.420075 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:15.616812 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:15.616782 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:16.198029 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:16.197996 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl"] Sep 11 15:36:16.245302 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:16.245269 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:16.439104 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:16.439066 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:21.589487 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:21.589451 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:36:24.696069 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:24.696033 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:31.217381 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.217321 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" podUID="0c3bc3c9-f277-4423-bed9-3e3f666c0fac" containerName="main" containerID="cri-o://a362ef507ecb050e3206c6542acb7367bbe33a3882ca9583f093704146059bb9" gracePeriod=45 Sep 11 15:36:31.419890 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.419858 2558 generic.go:356] "Generic (PLEG): container finished" podID="0c3bc3c9-f277-4423-bed9-3e3f666c0fac" containerID="a362ef507ecb050e3206c6542acb7367bbe33a3882ca9583f093704146059bb9" exitCode=0 Sep 11 15:36:31.420050 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.419932 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" event={"ID":"0c3bc3c9-f277-4423-bed9-3e3f666c0fac","Type":"ContainerDied","Data":"a362ef507ecb050e3206c6542acb7367bbe33a3882ca9583f093704146059bb9"} Sep 11 15:36:31.468251 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.468188 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:36:31.598671 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.598632 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-dshm\") pod \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " Sep 11 15:36:31.598885 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.598691 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-tmp-dir\") pod \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " Sep 11 15:36:31.598885 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.598764 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-tls-certs\") pod \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " Sep 11 15:36:31.598885 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.598805 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-home\") pod \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " Sep 11 15:36:31.598885 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.598864 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-model-cache\") pod \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " Sep 11 15:36:31.599136 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.598899 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm6vt\" (UniqueName: \"kubernetes.io/projected/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-kube-api-access-wm6vt\") pod \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\" (UID: \"0c3bc3c9-f277-4423-bed9-3e3f666c0fac\") " Sep 11 15:36:31.599136 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.598999 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c3bc3c9-f277-4423-bed9-3e3f666c0fac" (UID: "0c3bc3c9-f277-4423-bed9-3e3f666c0fac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:36:31.599136 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.599098 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c3bc3c9-f277-4423-bed9-3e3f666c0fac" (UID: "0c3bc3c9-f277-4423-bed9-3e3f666c0fac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:36:31.599303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.599228 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-home" (OuterVolumeSpecName: "home") pod "0c3bc3c9-f277-4423-bed9-3e3f666c0fac" (UID: "0c3bc3c9-f277-4423-bed9-3e3f666c0fac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:36:31.599303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.599289 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-tmp-dir\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:36:31.599303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.599303 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-model-cache\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:36:31.600966 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.600941 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-dshm" (OuterVolumeSpecName: "dshm") pod "0c3bc3c9-f277-4423-bed9-3e3f666c0fac" (UID: "0c3bc3c9-f277-4423-bed9-3e3f666c0fac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:36:31.601084 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.600984 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c3bc3c9-f277-4423-bed9-3e3f666c0fac" (UID: "0c3bc3c9-f277-4423-bed9-3e3f666c0fac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:36:31.601084 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.601005 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-kube-api-access-wm6vt" (OuterVolumeSpecName: "kube-api-access-wm6vt") pod "0c3bc3c9-f277-4423-bed9-3e3f666c0fac" (UID: "0c3bc3c9-f277-4423-bed9-3e3f666c0fac"). InnerVolumeSpecName "kube-api-access-wm6vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:36:31.700519 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.700492 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm6vt\" (UniqueName: \"kubernetes.io/projected/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-kube-api-access-wm6vt\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:36:31.700519 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.700517 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-dshm\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:36:31.700683 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.700527 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-tls-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:36:31.700683 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:31.700537 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3bc3c9-f277-4423-bed9-3e3f666c0fac-home\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:36:32.427314 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:32.427280 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" event={"ID":"0c3bc3c9-f277-4423-bed9-3e3f666c0fac","Type":"ContainerDied","Data":"31c075fac8ec8440f4d5835a042a25a33d44e3ad6ecd1c3b84d0c06a26447658"} Sep 11 15:36:32.427314 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:32.427292 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl" Sep 11 15:36:32.427779 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:32.427326 2558 scope.go:117] "RemoveContainer" containerID="a362ef507ecb050e3206c6542acb7367bbe33a3882ca9583f093704146059bb9" Sep 11 15:36:32.443807 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:32.443784 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl"] Sep 11 15:36:32.447209 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:32.447187 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-8559ee/ipp-route-8559ee-kserve-67cb5d65c9-jlhdl"] Sep 11 15:36:34.264360 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:34.264330 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c3bc3c9-f277-4423-bed9-3e3f666c0fac" path="/var/lib/kubelet/pods/0c3bc3c9-f277-4423-bed9-3e3f666c0fac/volumes" Sep 11 15:36:52.565825 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:52.565734 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:53.078062 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:53.078028 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:58.671051 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:58.671013 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:58.734391 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:58.734354 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:58.947512 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:58.947430 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst"] Sep 11 15:36:58.947710 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:58.947685 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerName="istio-proxy" containerID="cri-o://66b7b7bb1ac8b3c708afddb1328b34ee7c49e53c11cee318b73f75c1e7d14c39" gracePeriod=30 Sep 11 15:36:58.968234 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:58.968205 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:36:59.489931 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:36:59.489902 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:00.305714 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:00.305681 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:01.021026 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:01.020986 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:01.653879 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:01.653846 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:02.317041 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:02.317007 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:04.201883 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.201857 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:37:04.309145 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309057 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-credential-socket\") pod \"93e0eaca-1572-4be3-bb22-a0b471e06941\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " Sep 11 15:37:04.309145 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309092 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e0eaca-1572-4be3-bb22-a0b471e06941-istiod-ca-cert\") pod \"93e0eaca-1572-4be3-bb22-a0b471e06941\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " Sep 11 15:37:04.309145 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309111 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-workload-certs\") pod \"93e0eaca-1572-4be3-bb22-a0b471e06941\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " Sep 11 15:37:04.309377 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309159 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-workload-socket\") pod \"93e0eaca-1572-4be3-bb22-a0b471e06941\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " Sep 11 15:37:04.309377 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309277 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-data\") pod \"93e0eaca-1572-4be3-bb22-a0b471e06941\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " Sep 11 15:37:04.309377 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309335 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-envoy\") pod \"93e0eaca-1572-4be3-bb22-a0b471e06941\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " Sep 11 15:37:04.309539 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309374 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkfnx\" (UniqueName: \"kubernetes.io/projected/93e0eaca-1572-4be3-bb22-a0b471e06941-kube-api-access-dkfnx\") pod \"93e0eaca-1572-4be3-bb22-a0b471e06941\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " Sep 11 15:37:04.309539 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309416 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-token\") pod \"93e0eaca-1572-4be3-bb22-a0b471e06941\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " Sep 11 15:37:04.309633 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309412 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "93e0eaca-1572-4be3-bb22-a0b471e06941" (UID: "93e0eaca-1572-4be3-bb22-a0b471e06941"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:04.309633 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309543 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-data" (OuterVolumeSpecName: "istio-data") pod "93e0eaca-1572-4be3-bb22-a0b471e06941" (UID: "93e0eaca-1572-4be3-bb22-a0b471e06941"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:04.309633 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309421 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "93e0eaca-1572-4be3-bb22-a0b471e06941" (UID: "93e0eaca-1572-4be3-bb22-a0b471e06941"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:04.309633 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309542 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-podinfo\") pod \"93e0eaca-1572-4be3-bb22-a0b471e06941\" (UID: \"93e0eaca-1572-4be3-bb22-a0b471e06941\") " Sep 11 15:37:04.309854 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309524 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "93e0eaca-1572-4be3-bb22-a0b471e06941" (UID: "93e0eaca-1572-4be3-bb22-a0b471e06941"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:04.309854 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.309598 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e0eaca-1572-4be3-bb22-a0b471e06941-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "93e0eaca-1572-4be3-bb22-a0b471e06941" (UID: "93e0eaca-1572-4be3-bb22-a0b471e06941"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:37:04.310365 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.310059 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:04.310365 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.310083 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:04.310365 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.310098 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:04.310365 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.310110 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93e0eaca-1572-4be3-bb22-a0b471e06941-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:04.310365 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.310121 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:04.311844 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.311816 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "93e0eaca-1572-4be3-bb22-a0b471e06941" (UID: "93e0eaca-1572-4be3-bb22-a0b471e06941"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:04.311844 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.311821 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-token" (OuterVolumeSpecName: "istio-token") pod "93e0eaca-1572-4be3-bb22-a0b471e06941" (UID: "93e0eaca-1572-4be3-bb22-a0b471e06941"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:37:04.311975 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.311820 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e0eaca-1572-4be3-bb22-a0b471e06941-kube-api-access-dkfnx" (OuterVolumeSpecName: "kube-api-access-dkfnx") pod "93e0eaca-1572-4be3-bb22-a0b471e06941" (UID: "93e0eaca-1572-4be3-bb22-a0b471e06941"). InnerVolumeSpecName "kube-api-access-dkfnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:37:04.311975 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.311856 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "93e0eaca-1572-4be3-bb22-a0b471e06941" (UID: "93e0eaca-1572-4be3-bb22-a0b471e06941"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:37:04.411267 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.411228 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:04.411267 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.411260 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:04.411267 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.411271 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkfnx\" (UniqueName: \"kubernetes.io/projected/93e0eaca-1572-4be3-bb22-a0b471e06941-kube-api-access-dkfnx\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:04.411495 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.411280 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93e0eaca-1572-4be3-bb22-a0b471e06941-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:04.606940 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.606849 2558 generic.go:356] "Generic (PLEG): container finished" podID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerID="66b7b7bb1ac8b3c708afddb1328b34ee7c49e53c11cee318b73f75c1e7d14c39" exitCode=0 Sep 11 15:37:04.606940 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.606917 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" Sep 11 15:37:04.606940 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.606932 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" event={"ID":"93e0eaca-1572-4be3-bb22-a0b471e06941","Type":"ContainerDied","Data":"66b7b7bb1ac8b3c708afddb1328b34ee7c49e53c11cee318b73f75c1e7d14c39"} Sep 11 15:37:04.607173 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.606991 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst" event={"ID":"93e0eaca-1572-4be3-bb22-a0b471e06941","Type":"ContainerDied","Data":"37bee9ed6c8520d9d26d842a5b1ba818e98d513a0d07d9ab44a4cb795ae61622"} Sep 11 15:37:04.607173 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.607018 2558 scope.go:117] "RemoveContainer" containerID="66b7b7bb1ac8b3c708afddb1328b34ee7c49e53c11cee318b73f75c1e7d14c39" Sep 11 15:37:04.617096 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.617076 2558 scope.go:117] "RemoveContainer" containerID="c40535ef65103bf53d9f7c4bb421879fc7ea5d58db11d407b5b733d6e0fea353" Sep 11 15:37:04.626227 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.626205 2558 scope.go:117] "RemoveContainer" containerID="66b7b7bb1ac8b3c708afddb1328b34ee7c49e53c11cee318b73f75c1e7d14c39" Sep 11 15:37:04.626536 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:37:04.626516 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b7b7bb1ac8b3c708afddb1328b34ee7c49e53c11cee318b73f75c1e7d14c39\": container with ID starting with 66b7b7bb1ac8b3c708afddb1328b34ee7c49e53c11cee318b73f75c1e7d14c39 not found: ID does not exist" containerID="66b7b7bb1ac8b3c708afddb1328b34ee7c49e53c11cee318b73f75c1e7d14c39" Sep 11 15:37:04.626625 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.626544 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b7b7bb1ac8b3c708afddb1328b34ee7c49e53c11cee318b73f75c1e7d14c39"} err="failed to get container status \"66b7b7bb1ac8b3c708afddb1328b34ee7c49e53c11cee318b73f75c1e7d14c39\": rpc error: code = NotFound desc = could not find container \"66b7b7bb1ac8b3c708afddb1328b34ee7c49e53c11cee318b73f75c1e7d14c39\": container with ID starting with 66b7b7bb1ac8b3c708afddb1328b34ee7c49e53c11cee318b73f75c1e7d14c39 not found: ID does not exist" Sep 11 15:37:04.626625 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.626563 2558 scope.go:117] "RemoveContainer" containerID="c40535ef65103bf53d9f7c4bb421879fc7ea5d58db11d407b5b733d6e0fea353" Sep 11 15:37:04.626882 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:37:04.626863 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40535ef65103bf53d9f7c4bb421879fc7ea5d58db11d407b5b733d6e0fea353\": container with ID starting with c40535ef65103bf53d9f7c4bb421879fc7ea5d58db11d407b5b733d6e0fea353 not found: ID does not exist" containerID="c40535ef65103bf53d9f7c4bb421879fc7ea5d58db11d407b5b733d6e0fea353" Sep 11 15:37:04.626950 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.626887 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40535ef65103bf53d9f7c4bb421879fc7ea5d58db11d407b5b733d6e0fea353"} err="failed to get container status \"c40535ef65103bf53d9f7c4bb421879fc7ea5d58db11d407b5b733d6e0fea353\": rpc error: code = NotFound desc = could not find container \"c40535ef65103bf53d9f7c4bb421879fc7ea5d58db11d407b5b733d6e0fea353\": container with ID starting with c40535ef65103bf53d9f7c4bb421879fc7ea5d58db11d407b5b733d6e0fea353 not found: ID does not exist" Sep 11 15:37:04.628968 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.628946 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst"] Sep 11 15:37:04.632259 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:04.632224 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8559ee-openshift-default-b8f47945b-56dst"] Sep 11 15:37:06.265517 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:06.265484 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" path="/var/lib/kubelet/pods/93e0eaca-1572-4be3-bb22-a0b471e06941/volumes" Sep 11 15:37:09.959179 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:09.959143 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:10.019955 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:10.019919 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:10.076928 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:10.076894 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:10.145247 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:10.145211 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:10.202496 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:10.202457 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:10.533348 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:10.533314 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:10.635323 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:10.635282 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:11.295513 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:11.295481 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:12.025570 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:12.025537 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:13.385905 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:13.385868 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:37:16.245088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:16.245059 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:16.307773 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:16.307722 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:16.488233 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:16.488201 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf"] Sep 11 15:37:16.488455 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:16.488430 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" podUID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerName="istio-proxy" containerID="cri-o://6b04937e63dc1d932c4b05de9160bb7128ab414a247b2393643853aa30dd1477" gracePeriod=30 Sep 11 15:37:16.551806 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:16.551702 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:16.996510 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:16.996480 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:17.713938 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:17.713902 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:18.421317 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:18.421284 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:19.102587 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:19.102551 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:21.705120 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.705089 2558 generic.go:356] "Generic (PLEG): container finished" podID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerID="6b04937e63dc1d932c4b05de9160bb7128ab414a247b2393643853aa30dd1477" exitCode=0 Sep 11 15:37:21.705495 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.705161 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" event={"ID":"bfd22d1f-1195-4f3c-a007-542819ae4d1f","Type":"ContainerDied","Data":"6b04937e63dc1d932c4b05de9160bb7128ab414a247b2393643853aa30dd1477"} Sep 11 15:37:21.705495 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.705247 2558 scope.go:117] "RemoveContainer" containerID="9d71dd9dc62afec67f90a3361d7463e8806b92479ed2d88a64fe0721b26db395" Sep 11 15:37:21.752046 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.752015 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:37:21.878886 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.878802 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-podinfo\") pod \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " Sep 11 15:37:21.878886 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.878839 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-token\") pod \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " Sep 11 15:37:21.878886 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.878857 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istiod-ca-cert\") pod \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " Sep 11 15:37:21.878886 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.878887 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-envoy\") pod \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " Sep 11 15:37:21.879199 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.878921 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-data\") pod \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " Sep 11 15:37:21.879199 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.878976 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-credential-socket\") pod \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " Sep 11 15:37:21.879199 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.879032 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-workload-socket\") pod \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " Sep 11 15:37:21.879199 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.879047 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-workload-certs\") pod \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " Sep 11 15:37:21.879199 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.879073 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9xqw\" (UniqueName: \"kubernetes.io/projected/bfd22d1f-1195-4f3c-a007-542819ae4d1f-kube-api-access-t9xqw\") pod \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\" (UID: \"bfd22d1f-1195-4f3c-a007-542819ae4d1f\") " Sep 11 15:37:21.879454 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.879362 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bfd22d1f-1195-4f3c-a007-542819ae4d1f" (UID: "bfd22d1f-1195-4f3c-a007-542819ae4d1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:21.879454 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.879376 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bfd22d1f-1195-4f3c-a007-542819ae4d1f" (UID: "bfd22d1f-1195-4f3c-a007-542819ae4d1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:21.879454 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.879387 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bfd22d1f-1195-4f3c-a007-542819ae4d1f" (UID: "bfd22d1f-1195-4f3c-a007-542819ae4d1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:37:21.879615 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.879467 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-data" (OuterVolumeSpecName: "istio-data") pod "bfd22d1f-1195-4f3c-a007-542819ae4d1f" (UID: "bfd22d1f-1195-4f3c-a007-542819ae4d1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:21.879615 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.879587 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bfd22d1f-1195-4f3c-a007-542819ae4d1f" (UID: "bfd22d1f-1195-4f3c-a007-542819ae4d1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:21.881112 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.881088 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bfd22d1f-1195-4f3c-a007-542819ae4d1f" (UID: "bfd22d1f-1195-4f3c-a007-542819ae4d1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:37:21.881258 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.881237 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd22d1f-1195-4f3c-a007-542819ae4d1f-kube-api-access-t9xqw" (OuterVolumeSpecName: "kube-api-access-t9xqw") pod "bfd22d1f-1195-4f3c-a007-542819ae4d1f" (UID: "bfd22d1f-1195-4f3c-a007-542819ae4d1f"). InnerVolumeSpecName "kube-api-access-t9xqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:37:21.881487 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.881466 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bfd22d1f-1195-4f3c-a007-542819ae4d1f" (UID: "bfd22d1f-1195-4f3c-a007-542819ae4d1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:21.881534 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.881505 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-token" (OuterVolumeSpecName: "istio-token") pod "bfd22d1f-1195-4f3c-a007-542819ae4d1f" (UID: "bfd22d1f-1195-4f3c-a007-542819ae4d1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:37:21.980498 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.980457 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:21.980498 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.980490 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:21.980498 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.980501 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:21.980761 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.980511 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9xqw\" (UniqueName: \"kubernetes.io/projected/bfd22d1f-1195-4f3c-a007-542819ae4d1f-kube-api-access-t9xqw\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:21.980761 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.980525 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:21.980761 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.980535 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:21.980761 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.980543 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:21.980761 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.980551 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:21.980761 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:21.980559 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd22d1f-1195-4f3c-a007-542819ae4d1f-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:22.712068 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:22.712036 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" event={"ID":"bfd22d1f-1195-4f3c-a007-542819ae4d1f","Type":"ContainerDied","Data":"463bd4a1e94de97b16a734524290e269fe56fe4a315991be7312d2342281cc1f"} Sep 11 15:37:22.712068 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:22.712051 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf" Sep 11 15:37:22.712559 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:22.712095 2558 scope.go:117] "RemoveContainer" containerID="6b04937e63dc1d932c4b05de9160bb7128ab414a247b2393643853aa30dd1477" Sep 11 15:37:22.729345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:22.729304 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf"] Sep 11 15:37:22.732889 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:22.732865 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ce6e9-openshift-default-7fbc59cbdd-k7ljf"] Sep 11 15:37:24.264574 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:24.264543 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" path="/var/lib/kubelet/pods/bfd22d1f-1195-4f3c-a007-542819ae4d1f/volumes" Sep 11 15:37:27.397154 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:27.397118 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:27.462495 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:27.462459 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:27.561579 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:27.561548 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:27.621908 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:27.621877 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:27.939042 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:27.939005 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:28.039627 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:28.039594 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:28.707194 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:28.707164 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:30.483243 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:30.483208 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:37:33.850709 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:33.850678 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:33.915302 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:33.915264 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:34.082462 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:34.082429 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc"] Sep 11 15:37:34.082748 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:34.082713 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" containerID="cri-o://6eac7eaf51c5197eea045bec46542a3c95f603a2478d77fc38daec1f833ee994" gracePeriod=30 Sep 11 15:37:34.099603 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:34.099571 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:35.288350 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:35.288318 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:35.920528 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:35.920497 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:37.419492 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:37.419451 2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 15:37:37.419890 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:37.419522 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 15:37:39.333692 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.333663 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:37:39.429000 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.428968 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-token\") pod \"15ed46cf-24b8-450d-b140-21754d7bc97e\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " Sep 11 15:37:39.429209 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429029 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-data\") pod \"15ed46cf-24b8-450d-b140-21754d7bc97e\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " Sep 11 15:37:39.429209 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429062 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-podinfo\") pod \"15ed46cf-24b8-450d-b140-21754d7bc97e\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " Sep 11 15:37:39.429209 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429144 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-envoy\") pod \"15ed46cf-24b8-450d-b140-21754d7bc97e\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " Sep 11 15:37:39.429209 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429170 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-credential-socket\") pod \"15ed46cf-24b8-450d-b140-21754d7bc97e\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " Sep 11 15:37:39.429209 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429208 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-workload-socket\") pod \"15ed46cf-24b8-450d-b140-21754d7bc97e\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " Sep 11 15:37:39.429486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429244 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ed46cf-24b8-450d-b140-21754d7bc97e-istiod-ca-cert\") pod \"15ed46cf-24b8-450d-b140-21754d7bc97e\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " Sep 11 15:37:39.429486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429271 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-workload-certs\") pod \"15ed46cf-24b8-450d-b140-21754d7bc97e\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " Sep 11 15:37:39.429486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429315 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h6gq\" (UniqueName: \"kubernetes.io/projected/15ed46cf-24b8-450d-b140-21754d7bc97e-kube-api-access-8h6gq\") pod \"15ed46cf-24b8-450d-b140-21754d7bc97e\" (UID: \"15ed46cf-24b8-450d-b140-21754d7bc97e\") " Sep 11 15:37:39.429486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429412 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-data" (OuterVolumeSpecName: "istio-data") pod "15ed46cf-24b8-450d-b140-21754d7bc97e" (UID: "15ed46cf-24b8-450d-b140-21754d7bc97e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:39.429486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429464 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15ed46cf-24b8-450d-b140-21754d7bc97e" (UID: "15ed46cf-24b8-450d-b140-21754d7bc97e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:39.429721 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429583 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15ed46cf-24b8-450d-b140-21754d7bc97e" (UID: "15ed46cf-24b8-450d-b140-21754d7bc97e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:39.429721 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429676 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15ed46cf-24b8-450d-b140-21754d7bc97e" (UID: "15ed46cf-24b8-450d-b140-21754d7bc97e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:39.429721 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429697 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ed46cf-24b8-450d-b140-21754d7bc97e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15ed46cf-24b8-450d-b140-21754d7bc97e" (UID: "15ed46cf-24b8-450d-b140-21754d7bc97e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:37:39.429869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429721 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:39.429869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429761 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:39.429869 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.429777 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:39.431284 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.431256 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-token" (OuterVolumeSpecName: "istio-token") pod "15ed46cf-24b8-450d-b140-21754d7bc97e" (UID: "15ed46cf-24b8-450d-b140-21754d7bc97e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:37:39.431535 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.431516 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15ed46cf-24b8-450d-b140-21754d7bc97e" (UID: "15ed46cf-24b8-450d-b140-21754d7bc97e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:39.431758 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.431713 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15ed46cf-24b8-450d-b140-21754d7bc97e" (UID: "15ed46cf-24b8-450d-b140-21754d7bc97e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:37:39.431856 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.431781 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ed46cf-24b8-450d-b140-21754d7bc97e-kube-api-access-8h6gq" (OuterVolumeSpecName: "kube-api-access-8h6gq") pod "15ed46cf-24b8-450d-b140-21754d7bc97e" (UID: "15ed46cf-24b8-450d-b140-21754d7bc97e"). InnerVolumeSpecName "kube-api-access-8h6gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:37:39.530278 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.530238 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h6gq\" (UniqueName: \"kubernetes.io/projected/15ed46cf-24b8-450d-b140-21754d7bc97e-kube-api-access-8h6gq\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:39.530278 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.530273 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:39.530278 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.530282 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:39.530278 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.530290 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:39.530562 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.530299 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ed46cf-24b8-450d-b140-21754d7bc97e-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:39.530562 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.530307 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ed46cf-24b8-450d-b140-21754d7bc97e-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:39.813452 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.813369 2558 generic.go:356] "Generic (PLEG): container finished" podID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerID="6eac7eaf51c5197eea045bec46542a3c95f603a2478d77fc38daec1f833ee994" exitCode=0 Sep 11 15:37:39.813452 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.813404 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" event={"ID":"15ed46cf-24b8-450d-b140-21754d7bc97e","Type":"ContainerDied","Data":"6eac7eaf51c5197eea045bec46542a3c95f603a2478d77fc38daec1f833ee994"} Sep 11 15:37:39.813452 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.813417 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" Sep 11 15:37:39.813690 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.813486 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc" event={"ID":"15ed46cf-24b8-450d-b140-21754d7bc97e","Type":"ContainerDied","Data":"4d29189dc2168de8cdfc867cfce4e62208083580738646d5a72d100950eb4cd8"} Sep 11 15:37:39.813690 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.813511 2558 scope.go:117] "RemoveContainer" containerID="6eac7eaf51c5197eea045bec46542a3c95f603a2478d77fc38daec1f833ee994" Sep 11 15:37:39.823210 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.823192 2558 scope.go:117] "RemoveContainer" containerID="607dfcd28105639d6be82fa4e6b71266f755aa41b6d6e748b17187b19ed3ebbc" Sep 11 15:37:39.831278 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.831262 2558 scope.go:117] "RemoveContainer" containerID="6eac7eaf51c5197eea045bec46542a3c95f603a2478d77fc38daec1f833ee994" Sep 11 15:37:39.831543 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:37:39.831527 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eac7eaf51c5197eea045bec46542a3c95f603a2478d77fc38daec1f833ee994\": container with ID starting with 6eac7eaf51c5197eea045bec46542a3c95f603a2478d77fc38daec1f833ee994 not found: ID does not exist" containerID="6eac7eaf51c5197eea045bec46542a3c95f603a2478d77fc38daec1f833ee994" Sep 11 15:37:39.831590 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.831551 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eac7eaf51c5197eea045bec46542a3c95f603a2478d77fc38daec1f833ee994"} err="failed to get container status \"6eac7eaf51c5197eea045bec46542a3c95f603a2478d77fc38daec1f833ee994\": rpc error: code = NotFound desc = could not find container \"6eac7eaf51c5197eea045bec46542a3c95f603a2478d77fc38daec1f833ee994\": container with ID starting with 6eac7eaf51c5197eea045bec46542a3c95f603a2478d77fc38daec1f833ee994 not found: ID does not exist" Sep 11 15:37:39.831590 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.831567 2558 scope.go:117] "RemoveContainer" containerID="607dfcd28105639d6be82fa4e6b71266f755aa41b6d6e748b17187b19ed3ebbc" Sep 11 15:37:39.831812 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:37:39.831796 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607dfcd28105639d6be82fa4e6b71266f755aa41b6d6e748b17187b19ed3ebbc\": container with ID starting with 607dfcd28105639d6be82fa4e6b71266f755aa41b6d6e748b17187b19ed3ebbc not found: ID does not exist" containerID="607dfcd28105639d6be82fa4e6b71266f755aa41b6d6e748b17187b19ed3ebbc" Sep 11 15:37:39.831860 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.831817 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607dfcd28105639d6be82fa4e6b71266f755aa41b6d6e748b17187b19ed3ebbc"} err="failed to get container status \"607dfcd28105639d6be82fa4e6b71266f755aa41b6d6e748b17187b19ed3ebbc\": rpc error: code = NotFound desc = could not find container \"607dfcd28105639d6be82fa4e6b71266f755aa41b6d6e748b17187b19ed3ebbc\": container with ID starting with 607dfcd28105639d6be82fa4e6b71266f755aa41b6d6e748b17187b19ed3ebbc not found: ID does not exist" Sep 11 15:37:39.836238 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.836215 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc"] Sep 11 15:37:39.841816 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:39.841795 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-397e40-openshift-default-7c497bf9cb-z5mjc"] Sep 11 15:37:40.264734 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:40.264695 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" path="/var/lib/kubelet/pods/15ed46cf-24b8-450d-b140-21754d7bc97e/volumes" Sep 11 15:37:44.939662 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:44.939632 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:44.945263 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:44.945237 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt"] Sep 11 15:37:44.949391 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:44.949363 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-abba31-29819010-l7jgt"] Sep 11 15:37:45.042781 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:45.042720 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:45.140971 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:45.140941 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:45.207657 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:45.207577 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:45.495810 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:45.495717 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:45.727303 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:45.727256 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:46.263914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:46.263880 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2324a3a-892e-45c5-b20c-a4648e51f0ec" path="/var/lib/kubelet/pods/c2324a3a-892e-45c5-b20c-a4648e51f0ec/volumes" Sep 11 15:37:47.686014 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:47.685977 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:37:51.372487 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:51.372458 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:51.436624 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:51.436589 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:51.613148 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:51.613117 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594"] Sep 11 15:37:51.613358 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:51.613330 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerName="istio-proxy" containerID="cri-o://0f4b673f656c4072e9b138b0d8563546f30ba0bd6bc36f4f0ab473d04add38c5" gracePeriod=30 Sep 11 15:37:51.636793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:51.636704 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:53.915732 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:53.915692 2558 patch_prober.go:28] interesting pod/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 15:37:53.916101 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:53.915779 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 15:37:54.619890 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:54.619856 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:54.681104 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:54.681070 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:37:55.682950 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:55.682912 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:37:56.865413 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.865386 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:37:56.909894 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.909865 2558 generic.go:356] "Generic (PLEG): container finished" podID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerID="0f4b673f656c4072e9b138b0d8563546f30ba0bd6bc36f4f0ab473d04add38c5" exitCode=0 Sep 11 15:37:56.910088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.909913 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" Sep 11 15:37:56.910088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.909949 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" event={"ID":"cecda5b5-849e-4c16-9258-9a2fd54555ac","Type":"ContainerDied","Data":"0f4b673f656c4072e9b138b0d8563546f30ba0bd6bc36f4f0ab473d04add38c5"} Sep 11 15:37:56.910088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.910034 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594" event={"ID":"cecda5b5-849e-4c16-9258-9a2fd54555ac","Type":"ContainerDied","Data":"2153c58856e3e546ff9c9bd5fdf6772ec77a86a655c1781f14dd68e9822e7f21"} Sep 11 15:37:56.910088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.910066 2558 scope.go:117] "RemoveContainer" containerID="0f4b673f656c4072e9b138b0d8563546f30ba0bd6bc36f4f0ab473d04add38c5" Sep 11 15:37:56.918788 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.918774 2558 scope.go:117] "RemoveContainer" containerID="40e4e0ec98db5b3eb8bb7fa465facf1bca9a4d05867166537e77c091378b28fd" Sep 11 15:37:56.926583 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.926566 2558 scope.go:117] "RemoveContainer" containerID="0f4b673f656c4072e9b138b0d8563546f30ba0bd6bc36f4f0ab473d04add38c5" Sep 11 15:37:56.926898 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:37:56.926877 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4b673f656c4072e9b138b0d8563546f30ba0bd6bc36f4f0ab473d04add38c5\": container with ID starting with 0f4b673f656c4072e9b138b0d8563546f30ba0bd6bc36f4f0ab473d04add38c5 not found: ID does not exist" containerID="0f4b673f656c4072e9b138b0d8563546f30ba0bd6bc36f4f0ab473d04add38c5" Sep 11 15:37:56.926978 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.926910 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4b673f656c4072e9b138b0d8563546f30ba0bd6bc36f4f0ab473d04add38c5"} err="failed to get container status \"0f4b673f656c4072e9b138b0d8563546f30ba0bd6bc36f4f0ab473d04add38c5\": rpc error: code = NotFound desc = could not find container \"0f4b673f656c4072e9b138b0d8563546f30ba0bd6bc36f4f0ab473d04add38c5\": container with ID starting with 0f4b673f656c4072e9b138b0d8563546f30ba0bd6bc36f4f0ab473d04add38c5 not found: ID does not exist" Sep 11 15:37:56.926978 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.926933 2558 scope.go:117] "RemoveContainer" containerID="40e4e0ec98db5b3eb8bb7fa465facf1bca9a4d05867166537e77c091378b28fd" Sep 11 15:37:56.927184 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:37:56.927168 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e4e0ec98db5b3eb8bb7fa465facf1bca9a4d05867166537e77c091378b28fd\": container with ID starting with 40e4e0ec98db5b3eb8bb7fa465facf1bca9a4d05867166537e77c091378b28fd not found: ID does not exist" containerID="40e4e0ec98db5b3eb8bb7fa465facf1bca9a4d05867166537e77c091378b28fd" Sep 11 15:37:56.927244 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.927191 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e4e0ec98db5b3eb8bb7fa465facf1bca9a4d05867166537e77c091378b28fd"} err="failed to get container status \"40e4e0ec98db5b3eb8bb7fa465facf1bca9a4d05867166537e77c091378b28fd\": rpc error: code = NotFound desc = could not find container \"40e4e0ec98db5b3eb8bb7fa465facf1bca9a4d05867166537e77c091378b28fd\": container with ID starting with 40e4e0ec98db5b3eb8bb7fa465facf1bca9a4d05867166537e77c091378b28fd not found: ID does not exist" Sep 11 15:37:56.973250 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973226 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-credential-socket\") pod \"cecda5b5-849e-4c16-9258-9a2fd54555ac\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " Sep 11 15:37:56.973373 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973296 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-workload-certs\") pod \"cecda5b5-849e-4c16-9258-9a2fd54555ac\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " Sep 11 15:37:56.973373 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973314 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-podinfo\") pod \"cecda5b5-849e-4c16-9258-9a2fd54555ac\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " Sep 11 15:37:56.973373 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973332 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-data\") pod \"cecda5b5-849e-4c16-9258-9a2fd54555ac\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " Sep 11 15:37:56.973373 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973353 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-workload-socket\") pod \"cecda5b5-849e-4c16-9258-9a2fd54555ac\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " Sep 11 15:37:56.973556 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973383 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdh8w\" (UniqueName: \"kubernetes.io/projected/cecda5b5-849e-4c16-9258-9a2fd54555ac-kube-api-access-bdh8w\") pod \"cecda5b5-849e-4c16-9258-9a2fd54555ac\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " Sep 11 15:37:56.973556 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973428 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecda5b5-849e-4c16-9258-9a2fd54555ac-istiod-ca-cert\") pod \"cecda5b5-849e-4c16-9258-9a2fd54555ac\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " Sep 11 15:37:56.973556 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973465 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-token\") pod \"cecda5b5-849e-4c16-9258-9a2fd54555ac\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " Sep 11 15:37:56.973556 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973510 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-envoy\") pod \"cecda5b5-849e-4c16-9258-9a2fd54555ac\" (UID: \"cecda5b5-849e-4c16-9258-9a2fd54555ac\") " Sep 11 15:37:56.973556 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973521 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cecda5b5-849e-4c16-9258-9a2fd54555ac" (UID: "cecda5b5-849e-4c16-9258-9a2fd54555ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:56.973850 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973646 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "cecda5b5-849e-4c16-9258-9a2fd54555ac" (UID: "cecda5b5-849e-4c16-9258-9a2fd54555ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:56.973850 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973655 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cecda5b5-849e-4c16-9258-9a2fd54555ac" (UID: "cecda5b5-849e-4c16-9258-9a2fd54555ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:56.973850 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973670 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cecda5b5-849e-4c16-9258-9a2fd54555ac" (UID: "cecda5b5-849e-4c16-9258-9a2fd54555ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:56.973993 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973852 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-workload-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:56.973993 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973870 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-data\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:56.973993 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973881 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-workload-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:56.973993 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973893 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-credential-socket\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:56.973993 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.973941 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cecda5b5-849e-4c16-9258-9a2fd54555ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cecda5b5-849e-4c16-9258-9a2fd54555ac" (UID: "cecda5b5-849e-4c16-9258-9a2fd54555ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:37:56.975579 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.975551 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cecda5b5-849e-4c16-9258-9a2fd54555ac" (UID: "cecda5b5-849e-4c16-9258-9a2fd54555ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 15:37:56.975686 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.975553 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecda5b5-849e-4c16-9258-9a2fd54555ac-kube-api-access-bdh8w" (OuterVolumeSpecName: "kube-api-access-bdh8w") pod "cecda5b5-849e-4c16-9258-9a2fd54555ac" (UID: "cecda5b5-849e-4c16-9258-9a2fd54555ac"). InnerVolumeSpecName "kube-api-access-bdh8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:37:56.975686 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.975645 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cecda5b5-849e-4c16-9258-9a2fd54555ac" (UID: "cecda5b5-849e-4c16-9258-9a2fd54555ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:37:56.975800 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:56.975698 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "cecda5b5-849e-4c16-9258-9a2fd54555ac" (UID: "cecda5b5-849e-4c16-9258-9a2fd54555ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:37:57.074968 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:57.074937 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-podinfo\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:57.074968 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:57.074962 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdh8w\" (UniqueName: \"kubernetes.io/projected/cecda5b5-849e-4c16-9258-9a2fd54555ac-kube-api-access-bdh8w\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:57.074968 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:57.074972 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecda5b5-849e-4c16-9258-9a2fd54555ac-istiod-ca-cert\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:57.075198 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:57.074982 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-token\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:57.075198 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:57.074991 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecda5b5-849e-4c16-9258-9a2fd54555ac-istio-envoy\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:37:57.235666 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:57.235633 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594"] Sep 11 15:37:57.239487 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:57.239461 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abba31-openshift-default-6ff77b8d9d-mq594"] Sep 11 15:37:58.264313 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:37:58.264281 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" path="/var/lib/kubelet/pods/cecda5b5-849e-4c16-9258-9a2fd54555ac/volumes" Sep 11 15:38:26.572556 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:38:26.572527 2558 scope.go:117] "RemoveContainer" containerID="d57ef6f7793055d012d61986b40cdbc21ac64757553bd7cd4889469be4441cfe" Sep 11 15:38:26.582782 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:38:26.582729 2558 scope.go:117] "RemoveContainer" containerID="76bd125be1ec63a73c6601958a81593b4bbd67ff199dba4c07c8bfe3d447244a" Sep 11 15:38:52.860110 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:38:52.860073 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:39:15.309457 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:39:15.309415 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:39:15.688357 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:39:15.688325 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:39:17.047469 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:39:17.047434 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:39:41.094930 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:39:41.094893 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:39:49.286054 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:39:49.285974 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:39:52.542701 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:39:52.542665 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:39:57.084651 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:39:57.084619 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:40:14.484552 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:40:14.484500 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:40:20.401085 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:40:20.401051 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:41:47.993845 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:41:47.993807 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:41:53.985472 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:41:53.985433 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:42:01.583632 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:42:01.583595 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:42:01.981785 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:42:01.981750 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:42:09.180325 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:42:09.180282 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:42:11.321494 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:42:11.321461 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:42:19.985194 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:42:19.985153 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:42:26.640120 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:42:26.640087 2558 scope.go:117] "RemoveContainer" containerID="4c8e3962b0b4503e74f591ec09ec530c58732bc70e246ecc5f2cda8f759d9926" Sep 11 15:42:35.398120 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:42:35.398087 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:42:36.269511 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:42:36.269475 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:42:36.351635 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:42:36.351609 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:42:43.256868 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:42:43.256831 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:42:44.823897 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:42:44.823868 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:42:45.384906 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:42:45.384872 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:42:54.306883 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:42:54.306806 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:43:01.493530 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:01.493496 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:43:07.786698 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:07.786661 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:43:08.009406 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009354 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv"] Sep 11 15:43:08.009780 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009766 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerName="istio-proxy" Sep 11 15:43:08.009837 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009782 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerName="istio-proxy" Sep 11 15:43:08.009837 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009792 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerName="istio-proxy" Sep 11 15:43:08.009837 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009798 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerName="istio-proxy" Sep 11 15:43:08.009837 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009806 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerName="istio-proxy" Sep 11 15:43:08.009837 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009811 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerName="istio-proxy" Sep 11 15:43:08.009837 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009820 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" Sep 11 15:43:08.009837 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009827 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" Sep 11 15:43:08.009837 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009836 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009843 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009855 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009860 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009870 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009875 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009881 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb" containerName="revoke-keys" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009886 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb" containerName="revoke-keys" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009891 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009897 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009913 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c3bc3c9-f277-4423-bed9-3e3f666c0fac" containerName="main" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009920 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c3bc3c9-f277-4423-bed9-3e3f666c0fac" containerName="main" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009975 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009983 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="15ed46cf-24b8-450d-b140-21754d7bc97e" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.009994 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.010000 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e80a24a3-f83e-4e57-8a0c-a8a384b2c7bb" containerName="revoke-keys" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.010005 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.010011 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cecda5b5-849e-4c16-9258-9a2fd54555ac" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.010016 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerName="istio-proxy" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.010023 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c3bc3c9-f277-4423-bed9-3e3f666c0fac" containerName="main" Sep 11 15:43:08.010087 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.010029 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e0eaca-1572-4be3-bb22-a0b471e06941" containerName="istio-proxy" Sep 11 15:43:08.013126 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.013111 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.016423 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.016398 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-79rjx\"" Sep 11 15:43:08.016559 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.016422 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 15:43:08.016559 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.016459 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 15:43:08.016559 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.016476 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 15:43:08.016559 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.016398 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 15:43:08.025130 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.025110 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv"] Sep 11 15:43:08.040658 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.040586 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3e2340-b3ba-493a-becb-b0419303257e-maas-api-tls\") pod \"maas-api-6b6fb88c84-hjzgv\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.040793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.040677 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3e2340-b3ba-493a-becb-b0419303257e-openshift-service-ca\") pod \"maas-api-6b6fb88c84-hjzgv\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.040793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.040704 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhxr\" (UniqueName: \"kubernetes.io/projected/ea3e2340-b3ba-493a-becb-b0419303257e-kube-api-access-5zhxr\") pod \"maas-api-6b6fb88c84-hjzgv\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.040793 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.040772 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea3e2340-b3ba-493a-becb-b0419303257e-metrics-certs\") pod \"maas-api-6b6fb88c84-hjzgv\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.141795 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.141760 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3e2340-b3ba-493a-becb-b0419303257e-maas-api-tls\") pod \"maas-api-6b6fb88c84-hjzgv\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.141952 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.141803 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3e2340-b3ba-493a-becb-b0419303257e-openshift-service-ca\") pod \"maas-api-6b6fb88c84-hjzgv\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.141952 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.141827 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhxr\" (UniqueName: \"kubernetes.io/projected/ea3e2340-b3ba-493a-becb-b0419303257e-kube-api-access-5zhxr\") pod \"maas-api-6b6fb88c84-hjzgv\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.141952 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.141851 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea3e2340-b3ba-493a-becb-b0419303257e-metrics-certs\") pod \"maas-api-6b6fb88c84-hjzgv\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.142563 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.142537 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3e2340-b3ba-493a-becb-b0419303257e-openshift-service-ca\") pod \"maas-api-6b6fb88c84-hjzgv\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.144219 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.144191 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3e2340-b3ba-493a-becb-b0419303257e-maas-api-tls\") pod \"maas-api-6b6fb88c84-hjzgv\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.144365 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.144344 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea3e2340-b3ba-493a-becb-b0419303257e-metrics-certs\") pod \"maas-api-6b6fb88c84-hjzgv\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.149696 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.149672 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhxr\" (UniqueName: \"kubernetes.io/projected/ea3e2340-b3ba-493a-becb-b0419303257e-kube-api-access-5zhxr\") pod \"maas-api-6b6fb88c84-hjzgv\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.325010 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.324943 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:08.597290 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.597210 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv"] Sep 11 15:43:08.599920 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:43:08.599887 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3e2340_b3ba_493a_becb_b0419303257e.slice/crio-583632c5a7a62a7789c92e8f9548330961e97e81483477c014ab4ee2bd16a51c WatchSource:0}: Error finding container 583632c5a7a62a7789c92e8f9548330961e97e81483477c014ab4ee2bd16a51c: Status 404 returned error can't find the container with id 583632c5a7a62a7789c92e8f9548330961e97e81483477c014ab4ee2bd16a51c Sep 11 15:43:08.601240 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:08.601217 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 15:43:09.536147 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:09.536107 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" event={"ID":"ea3e2340-b3ba-493a-becb-b0419303257e","Type":"ContainerStarted","Data":"5efee844c6fe494d8994382999bade5d59054ad27ba87afc45596b0a708e8231"} Sep 11 15:43:09.536562 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:09.536171 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" event={"ID":"ea3e2340-b3ba-493a-becb-b0419303257e","Type":"ContainerStarted","Data":"583632c5a7a62a7789c92e8f9548330961e97e81483477c014ab4ee2bd16a51c"} Sep 11 15:43:09.536562 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:09.536336 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:09.557084 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:09.557043 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" podStartSLOduration=2.120726855 podStartE2EDuration="2.55703008s" totalImagesPullingTime="436.303225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:43:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:43:09.554534925 +0000 UTC m=+2083.786459373" watchObservedRunningTime="2026-09-11 15:43:09.55703008 +0000 UTC m=+2083.788954511" Sep 11 15:43:15.346483 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.346438 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:43:15.490914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.490883 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv"] Sep 11 15:43:15.491123 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.491097 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" podUID="ea3e2340-b3ba-493a-becb-b0419303257e" containerName="maas-api" containerID="cri-o://5efee844c6fe494d8994382999bade5d59054ad27ba87afc45596b0a708e8231" gracePeriod=30 Sep 11 15:43:15.496195 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.496173 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:15.528585 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.528561 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m"] Sep 11 15:43:15.528965 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.528952 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfd22d1f-1195-4f3c-a007-542819ae4d1f" containerName="istio-proxy" Sep 11 15:43:15.531954 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.531937 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.540654 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.540629 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m"] Sep 11 15:43:15.605224 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.605166 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8j67\" (UniqueName: \"kubernetes.io/projected/364262d8-45e8-4a6c-b232-46bc7d902909-kube-api-access-d8j67\") pod \"maas-api-cd89654bd-pdc5m\" (UID: \"364262d8-45e8-4a6c-b232-46bc7d902909\") " pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.605224 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.605208 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/364262d8-45e8-4a6c-b232-46bc7d902909-openshift-service-ca\") pod \"maas-api-cd89654bd-pdc5m\" (UID: \"364262d8-45e8-4a6c-b232-46bc7d902909\") " pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.605364 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.605226 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364262d8-45e8-4a6c-b232-46bc7d902909-maas-api-tls\") pod \"maas-api-cd89654bd-pdc5m\" (UID: \"364262d8-45e8-4a6c-b232-46bc7d902909\") " pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.605364 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.605260 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364262d8-45e8-4a6c-b232-46bc7d902909-metrics-certs\") pod \"maas-api-cd89654bd-pdc5m\" (UID: \"364262d8-45e8-4a6c-b232-46bc7d902909\") " pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.706288 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.706253 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8j67\" (UniqueName: \"kubernetes.io/projected/364262d8-45e8-4a6c-b232-46bc7d902909-kube-api-access-d8j67\") pod \"maas-api-cd89654bd-pdc5m\" (UID: \"364262d8-45e8-4a6c-b232-46bc7d902909\") " pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.706434 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.706313 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/364262d8-45e8-4a6c-b232-46bc7d902909-openshift-service-ca\") pod \"maas-api-cd89654bd-pdc5m\" (UID: \"364262d8-45e8-4a6c-b232-46bc7d902909\") " pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.706434 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.706344 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364262d8-45e8-4a6c-b232-46bc7d902909-maas-api-tls\") pod \"maas-api-cd89654bd-pdc5m\" (UID: \"364262d8-45e8-4a6c-b232-46bc7d902909\") " pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.706434 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.706362 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364262d8-45e8-4a6c-b232-46bc7d902909-metrics-certs\") pod \"maas-api-cd89654bd-pdc5m\" (UID: \"364262d8-45e8-4a6c-b232-46bc7d902909\") " pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.706968 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.706943 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/364262d8-45e8-4a6c-b232-46bc7d902909-openshift-service-ca\") pod \"maas-api-cd89654bd-pdc5m\" (UID: \"364262d8-45e8-4a6c-b232-46bc7d902909\") " pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.708922 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.708899 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364262d8-45e8-4a6c-b232-46bc7d902909-maas-api-tls\") pod \"maas-api-cd89654bd-pdc5m\" (UID: \"364262d8-45e8-4a6c-b232-46bc7d902909\") " pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.708922 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.708913 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364262d8-45e8-4a6c-b232-46bc7d902909-metrics-certs\") pod \"maas-api-cd89654bd-pdc5m\" (UID: \"364262d8-45e8-4a6c-b232-46bc7d902909\") " pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.715634 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.715609 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8j67\" (UniqueName: \"kubernetes.io/projected/364262d8-45e8-4a6c-b232-46bc7d902909-kube-api-access-d8j67\") pod \"maas-api-cd89654bd-pdc5m\" (UID: \"364262d8-45e8-4a6c-b232-46bc7d902909\") " pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.733345 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.733322 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:15.807130 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.807095 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3e2340-b3ba-493a-becb-b0419303257e-maas-api-tls\") pod \"ea3e2340-b3ba-493a-becb-b0419303257e\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " Sep 11 15:43:15.807290 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.807181 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea3e2340-b3ba-493a-becb-b0419303257e-metrics-certs\") pod \"ea3e2340-b3ba-493a-becb-b0419303257e\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " Sep 11 15:43:15.807290 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.807203 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zhxr\" (UniqueName: \"kubernetes.io/projected/ea3e2340-b3ba-493a-becb-b0419303257e-kube-api-access-5zhxr\") pod \"ea3e2340-b3ba-493a-becb-b0419303257e\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " Sep 11 15:43:15.807290 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.807235 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3e2340-b3ba-493a-becb-b0419303257e-openshift-service-ca\") pod \"ea3e2340-b3ba-493a-becb-b0419303257e\" (UID: \"ea3e2340-b3ba-493a-becb-b0419303257e\") " Sep 11 15:43:15.807649 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.807624 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3e2340-b3ba-493a-becb-b0419303257e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea3e2340-b3ba-493a-becb-b0419303257e" (UID: "ea3e2340-b3ba-493a-becb-b0419303257e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:43:15.809213 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.809185 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3e2340-b3ba-493a-becb-b0419303257e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea3e2340-b3ba-493a-becb-b0419303257e" (UID: "ea3e2340-b3ba-493a-becb-b0419303257e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:43:15.809306 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.809256 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3e2340-b3ba-493a-becb-b0419303257e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ea3e2340-b3ba-493a-becb-b0419303257e" (UID: "ea3e2340-b3ba-493a-becb-b0419303257e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:43:15.809366 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.809317 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3e2340-b3ba-493a-becb-b0419303257e-kube-api-access-5zhxr" (OuterVolumeSpecName: "kube-api-access-5zhxr") pod "ea3e2340-b3ba-493a-becb-b0419303257e" (UID: "ea3e2340-b3ba-493a-becb-b0419303257e"). InnerVolumeSpecName "kube-api-access-5zhxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:43:15.876569 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.876508 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:15.908651 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.908620 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3e2340-b3ba-493a-becb-b0419303257e-openshift-service-ca\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:43:15.908651 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.908646 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3e2340-b3ba-493a-becb-b0419303257e-maas-api-tls\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:43:15.908651 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.908656 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea3e2340-b3ba-493a-becb-b0419303257e-metrics-certs\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:43:15.908935 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:15.908664 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zhxr\" (UniqueName: \"kubernetes.io/projected/ea3e2340-b3ba-493a-becb-b0419303257e-kube-api-access-5zhxr\") on node \"ip-10-0-134-128.ec2.internal\" DevicePath \"\"" Sep 11 15:43:16.148402 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.148376 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m"] Sep 11 15:43:16.150055 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:43:16.150029 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364262d8_45e8_4a6c_b232_46bc7d902909.slice/crio-6dac6905c8ea1b2ac39a7b6fcf97ea67eddbc71fb70da656383ff07ebd528747 WatchSource:0}: Error finding container 6dac6905c8ea1b2ac39a7b6fcf97ea67eddbc71fb70da656383ff07ebd528747: Status 404 returned error can't find the container with id 6dac6905c8ea1b2ac39a7b6fcf97ea67eddbc71fb70da656383ff07ebd528747 Sep 11 15:43:16.573572 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.573533 2558 generic.go:356] "Generic (PLEG): container finished" podID="ea3e2340-b3ba-493a-becb-b0419303257e" containerID="5efee844c6fe494d8994382999bade5d59054ad27ba87afc45596b0a708e8231" exitCode=0 Sep 11 15:43:16.573994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.573581 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" Sep 11 15:43:16.573994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.573613 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" event={"ID":"ea3e2340-b3ba-493a-becb-b0419303257e","Type":"ContainerDied","Data":"5efee844c6fe494d8994382999bade5d59054ad27ba87afc45596b0a708e8231"} Sep 11 15:43:16.573994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.573665 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv" event={"ID":"ea3e2340-b3ba-493a-becb-b0419303257e","Type":"ContainerDied","Data":"583632c5a7a62a7789c92e8f9548330961e97e81483477c014ab4ee2bd16a51c"} Sep 11 15:43:16.573994 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.573688 2558 scope.go:117] "RemoveContainer" containerID="5efee844c6fe494d8994382999bade5d59054ad27ba87afc45596b0a708e8231" Sep 11 15:43:16.575945 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.575776 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" event={"ID":"364262d8-45e8-4a6c-b232-46bc7d902909","Type":"ContainerStarted","Data":"98782937fdd41e27f51ecf7dfb587e86af02bb9713ac7607a6174bf7fe69d1dc"} Sep 11 15:43:16.575945 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.575865 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" event={"ID":"364262d8-45e8-4a6c-b232-46bc7d902909","Type":"ContainerStarted","Data":"6dac6905c8ea1b2ac39a7b6fcf97ea67eddbc71fb70da656383ff07ebd528747"} Sep 11 15:43:16.576117 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.575960 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:16.582735 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.582713 2558 scope.go:117] "RemoveContainer" containerID="5efee844c6fe494d8994382999bade5d59054ad27ba87afc45596b0a708e8231" Sep 11 15:43:16.583043 ip-10-0-134-128 kubenswrapper[2558]: E0911 15:43:16.583003 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5efee844c6fe494d8994382999bade5d59054ad27ba87afc45596b0a708e8231\": container with ID starting with 5efee844c6fe494d8994382999bade5d59054ad27ba87afc45596b0a708e8231 not found: ID does not exist" containerID="5efee844c6fe494d8994382999bade5d59054ad27ba87afc45596b0a708e8231" Sep 11 15:43:16.583125 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.583037 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efee844c6fe494d8994382999bade5d59054ad27ba87afc45596b0a708e8231"} err="failed to get container status \"5efee844c6fe494d8994382999bade5d59054ad27ba87afc45596b0a708e8231\": rpc error: code = NotFound desc = could not find container \"5efee844c6fe494d8994382999bade5d59054ad27ba87afc45596b0a708e8231\": container with ID starting with 5efee844c6fe494d8994382999bade5d59054ad27ba87afc45596b0a708e8231 not found: ID does not exist" Sep 11 15:43:16.590966 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.590946 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv"] Sep 11 15:43:16.593082 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.593065 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6fb88c84-hjzgv"] Sep 11 15:43:16.606819 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:16.606782 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" podStartSLOduration=1.2928330080000001 podStartE2EDuration="1.606770447s" totalImagesPullingTime="313.937439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:43:16.606084414 +0000 UTC m=+2090.838008847" watchObservedRunningTime="2026-09-11 15:43:16.606770447 +0000 UTC m=+2090.838694891" Sep 11 15:43:18.264714 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:18.264679 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea3e2340-b3ba-493a-becb-b0419303257e" path="/var/lib/kubelet/pods/ea3e2340-b3ba-493a-becb-b0419303257e/volumes" Sep 11 15:43:19.484828 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:19.484791 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:43:21.895870 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:21.895831 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:43:22.587549 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:22.587517 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-cd89654bd-pdc5m" Sep 11 15:43:27.394088 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:27.394058 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:43:34.493523 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:34.493486 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-sxmbv"] Sep 11 15:43:43.577794 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:43.577764 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-s5g46_b44c03e3-cd6a-41ee-9233-777271b72b2c/server/0.log" Sep 11 15:43:43.698269 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:43.698242 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-62ngs_2ecd26b7-a4e6-4ae1-8485-3a619d964390/manager/0.log" Sep 11 15:43:43.806279 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:43.806251 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-87954dbdf-6jmfm_2852e3fd-6050-427d-a82a-9c573aaa21e3/copy-manifests/0.log" Sep 11 15:43:43.811872 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:43.811851 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-87954dbdf-6jmfm_2852e3fd-6050-427d-a82a-9c573aaa21e3/manager/0.log" Sep 11 15:43:44.522735 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:44.522706 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-cd89654bd-pdc5m_364262d8-45e8-4a6c-b232-46bc7d902909/maas-api/0.log" Sep 11 15:43:44.639754 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:44.639706 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819010-jph5p_1f3f8956-5bb1-4ef3-a3f5-aed47b6fcbe2/cleanup/0.log" Sep 11 15:43:46.078809 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:46.078784 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-sxmbv_9676ad60-14ee-411a-a65d-aaff7ed1cf11/limitador/0.log" Sep 11 15:43:46.652325 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:46.652296 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4slk8_c9ba6cbd-6e02-4ef4-aeb7-989bb863027d/discovery/0.log" Sep 11 15:43:47.949034 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:47.949005 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-5whpz_242f596d-f635-43c0-bc17-3bfdd232eea4/main/0.log" Sep 11 15:43:48.165279 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:48.165252 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftz6dl_5b0a4329-dfd8-4944-9f1b-9a0b53698c1a/main/0.log" Sep 11 15:43:53.668493 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:53.668460 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:43:53.809209 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:53.809176 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" limit=1024 Sep 11 15:43:56.528310 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:56.528283 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cdnlq_f547eaf0-def0-4524-9755-e682cd106e33/global-pull-secret-syncer/0.log" Sep 11 15:43:56.577140 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:56.577114 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5hxt6_853552d8-4997-4150-a85f-ac633b3fa6bd/konnectivity-agent/0.log" Sep 11 15:43:56.665756 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:43:56.665714 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-128.ec2.internal_4e1779d60c22f5ff6cb06e66ddefda83/haproxy/0.log" Sep 11 15:44:01.835914 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:01.835887 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-sxmbv_9676ad60-14ee-411a-a65d-aaff7ed1cf11/limitador/0.log" Sep 11 15:44:03.577294 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:03.577261 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-755d4b68f4-4vhnv_f23ecfa6-77ec-4278-b534-84916a0d33e1/metrics-server/0.log" Sep 11 15:44:03.644229 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:03.644197 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48qcs_ac7a7cd7-2c65-443a-85a2-b7da044b1a53/node-exporter/0.log" Sep 11 15:44:03.660031 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:03.660009 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48qcs_ac7a7cd7-2c65-443a-85a2-b7da044b1a53/kube-rbac-proxy/0.log" Sep 11 15:44:03.683072 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:03.683053 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48qcs_ac7a7cd7-2c65-443a-85a2-b7da044b1a53/init-textfile/0.log" Sep 11 15:44:04.228288 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:04.228256 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-cpqn7_ea38a4c7-6e87-4e46-ac15-a530c206f519/prometheus-operator/0.log" Sep 11 15:44:04.267148 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:04.267124 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-cpqn7_ea38a4c7-6e87-4e46-ac15-a530c206f519/kube-rbac-proxy/0.log" Sep 11 15:44:04.313121 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:04.313102 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-28bqs_e41e189f-4474-45ff-acc7-5c7cadf587ed/prometheus-operator-admission-webhook/0.log" Sep 11 15:44:04.369280 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:04.369256 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b5d5cd569-9v555_661a06de-e120-4fcb-816d-90b4941120e8/telemeter-client/0.log" Sep 11 15:44:04.407159 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:04.407135 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b5d5cd569-9v555_661a06de-e120-4fcb-816d-90b4941120e8/reload/0.log" Sep 11 15:44:04.453207 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:04.453184 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b5d5cd569-9v555_661a06de-e120-4fcb-816d-90b4941120e8/kube-rbac-proxy/0.log" Sep 11 15:44:05.902973 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:05.902943 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-zk9nm_c1a738b7-2dd6-4672-a3d1-866ff1a01b15/networking-console-plugin/0.log" Sep 11 15:44:06.435991 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.435958 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl"] Sep 11 15:44:06.436329 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.436315 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea3e2340-b3ba-493a-becb-b0419303257e" containerName="maas-api" Sep 11 15:44:06.436380 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.436332 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea3e2340-b3ba-493a-becb-b0419303257e" containerName="maas-api" Sep 11 15:44:06.436416 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.436394 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea3e2340-b3ba-493a-becb-b0419303257e" containerName="maas-api" Sep 11 15:44:06.439519 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.439498 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.442677 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.442651 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c7pwf\"/\"default-dockercfg-5nlz8\"" Sep 11 15:44:06.442812 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.442654 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c7pwf\"/\"kube-root-ca.crt\"" Sep 11 15:44:06.442812 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.442704 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c7pwf\"/\"openshift-service-ca.crt\"" Sep 11 15:44:06.452850 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.452829 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl"] Sep 11 15:44:06.547642 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.547611 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76263788-8194-40e5-8c7c-e870d6333f11-lib-modules\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.547834 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.547658 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv4z7\" (UniqueName: \"kubernetes.io/projected/76263788-8194-40e5-8c7c-e870d6333f11-kube-api-access-wv4z7\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.547834 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.547803 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/76263788-8194-40e5-8c7c-e870d6333f11-podres\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.547976 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.547887 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76263788-8194-40e5-8c7c-e870d6333f11-sys\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.547976 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.547907 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/76263788-8194-40e5-8c7c-e870d6333f11-proc\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.649041 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.649011 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/76263788-8194-40e5-8c7c-e870d6333f11-podres\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.649172 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.649056 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76263788-8194-40e5-8c7c-e870d6333f11-sys\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.649172 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.649071 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/76263788-8194-40e5-8c7c-e870d6333f11-proc\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.649172 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.649112 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76263788-8194-40e5-8c7c-e870d6333f11-lib-modules\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.649172 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.649146 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv4z7\" (UniqueName: \"kubernetes.io/projected/76263788-8194-40e5-8c7c-e870d6333f11-kube-api-access-wv4z7\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.649172 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.649153 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76263788-8194-40e5-8c7c-e870d6333f11-sys\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.649172 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.649168 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/76263788-8194-40e5-8c7c-e870d6333f11-podres\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.649412 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.649220 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/76263788-8194-40e5-8c7c-e870d6333f11-proc\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.649412 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.649268 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76263788-8194-40e5-8c7c-e870d6333f11-lib-modules\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.661647 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.661630 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv4z7\" (UniqueName: \"kubernetes.io/projected/76263788-8194-40e5-8c7c-e870d6333f11-kube-api-access-wv4z7\") pod \"perf-node-gather-daemonset-n4cwl\" (UID: \"76263788-8194-40e5-8c7c-e870d6333f11\") " pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:06.749550 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:06.749483 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:07.024432 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:07.024399 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl"] Sep 11 15:44:07.025879 ip-10-0-134-128 kubenswrapper[2558]: W0911 15:44:07.025853 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod76263788_8194_40e5_8c7c_e870d6333f11.slice/crio-6b9c96d83c9b247a7df3f9950c0001ed65470497934dc7c2b101d0474dbecccf WatchSource:0}: Error finding container 6b9c96d83c9b247a7df3f9950c0001ed65470497934dc7c2b101d0474dbecccf: Status 404 returned error can't find the container with id 6b9c96d83c9b247a7df3f9950c0001ed65470497934dc7c2b101d0474dbecccf Sep 11 15:44:07.192676 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:07.192652 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-52rkt_9fa10168-da0c-4e3f-aaf6-73db67dd6227/download-server/0.log" Sep 11 15:44:07.849583 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:07.849550 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" event={"ID":"76263788-8194-40e5-8c7c-e870d6333f11","Type":"ContainerStarted","Data":"af054e25f9cc0ccc5316b694f769a862c5caa3622f0a8b091d386650a7f4dc66"} Sep 11 15:44:07.849783 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:07.849678 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:07.849783 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:07.849702 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" event={"ID":"76263788-8194-40e5-8c7c-e870d6333f11","Type":"ContainerStarted","Data":"6b9c96d83c9b247a7df3f9950c0001ed65470497934dc7c2b101d0474dbecccf"} Sep 11 15:44:07.871232 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:07.871164 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" podStartSLOduration=1.87114912 podStartE2EDuration="1.87114912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:44:07.869943766 +0000 UTC m=+2142.101868197" watchObservedRunningTime="2026-09-11 15:44:07.87114912 +0000 UTC m=+2142.103073551" Sep 11 15:44:09.000820 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:09.000790 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tngb6_584cffdd-40bc-4fd9-bd8f-fb92dd86978e/dns/0.log" Sep 11 15:44:09.024754 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:09.024711 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tngb6_584cffdd-40bc-4fd9-bd8f-fb92dd86978e/kube-rbac-proxy/0.log" Sep 11 15:44:09.069668 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:09.069641 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r85xv_d209e966-9242-4f9d-8d2d-63dc61ffc5c3/dns-node-resolver/0.log" Sep 11 15:44:09.631497 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:09.631463 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4mzss_6ddcf6d3-17a1-4b8a-b563-dd7be8640bcb/node-ca/0.log" Sep 11 15:44:10.990319 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:10.990281 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4slk8_c9ba6cbd-6e02-4ef4-aeb7-989bb863027d/discovery/0.log" Sep 11 15:44:12.024829 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:12.024801 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dp7jn_379aa0e8-9d7d-42dc-bd0d-acb425c71a2d/serve-healthcheck-canary/0.log" Sep 11 15:44:12.612846 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:12.612812 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8xm5j_69574f6d-1b8f-4d63-b24d-119756a0267d/kube-rbac-proxy/0.log" Sep 11 15:44:12.635275 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:12.635246 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8xm5j_69574f6d-1b8f-4d63-b24d-119756a0267d/exporter/0.log" Sep 11 15:44:12.665177 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:12.665155 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8xm5j_69574f6d-1b8f-4d63-b24d-119756a0267d/extractor/0.log" Sep 11 15:44:13.865378 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:13.865351 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c7pwf/perf-node-gather-daemonset-n4cwl" Sep 11 15:44:15.277271 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:15.277203 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-s5g46_b44c03e3-cd6a-41ee-9233-777271b72b2c/server/0.log" Sep 11 15:44:15.359895 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:15.359874 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-62ngs_2ecd26b7-a4e6-4ae1-8485-3a619d964390/manager/0.log" Sep 11 15:44:15.383727 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:15.383701 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-87954dbdf-6jmfm_2852e3fd-6050-427d-a82a-9c573aaa21e3/manager/0.log" Sep 11 15:44:15.407021 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:15.406994 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-87954dbdf-6jmfm_2852e3fd-6050-427d-a82a-9c573aaa21e3/copy-manifests/0.log" Sep 11 15:44:16.939452 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:16.939423 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68f4698bc6-vs7m4_fe914ef9-a2c8-4955-ac2e-0dd01e15ee91/manager/0.log" Sep 11 15:44:16.965310 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:16.965287 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-kmng8_9af099e5-5c97-4000-bfe4-bff57a6b47a8/openshift-lws-operator/0.log" Sep 11 15:44:17.167607 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:17.167577 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4slk8_c9ba6cbd-6e02-4ef4-aeb7-989bb863027d/discovery/0.log" Sep 11 15:44:19.145486 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:19.145411 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4slk8_c9ba6cbd-6e02-4ef4-aeb7-989bb863027d/discovery/0.log" Sep 11 15:44:22.507321 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:22.507243 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-xdjlq_598ebdf6-3cba-4a77-b9ab-b8edaee3db5f/kube-storage-version-migrator-operator/1.log" Sep 11 15:44:22.508110 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:22.508091 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-xdjlq_598ebdf6-3cba-4a77-b9ab-b8edaee3db5f/kube-storage-version-migrator-operator/0.log" Sep 11 15:44:23.458846 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:23.458815 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6b84v_ac2f5397-4638-4694-9857-e5a25f186318/kube-multus-additional-cni-plugins/0.log" Sep 11 15:44:23.479351 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:23.479319 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6b84v_ac2f5397-4638-4694-9857-e5a25f186318/egress-router-binary-copy/0.log" Sep 11 15:44:23.504972 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:23.504948 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6b84v_ac2f5397-4638-4694-9857-e5a25f186318/cni-plugins/0.log" Sep 11 15:44:23.525523 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:23.525502 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6b84v_ac2f5397-4638-4694-9857-e5a25f186318/bond-cni-plugin/0.log" Sep 11 15:44:23.549132 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:23.549111 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6b84v_ac2f5397-4638-4694-9857-e5a25f186318/routeoverride-cni/0.log" Sep 11 15:44:23.578113 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:23.578093 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6b84v_ac2f5397-4638-4694-9857-e5a25f186318/whereabouts-cni-bincopy/0.log" Sep 11 15:44:23.600058 ip-10-0-134-128 kubenswrapper[2558]: I0911 15:44:23.600041 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6b84v_ac2f5397-4638-4694-9857-e5a25f186318/whereabouts-cni/0.log"