Sep 18 17:14:45.220716 ip-10-0-128-9 systemd[1]: Starting Kubernetes Kubelet... Sep 18 17:14:48.685269 ip-10-0-128-9 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 17:14:48.685269 ip-10-0-128-9 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 18 17:14:48.685269 ip-10-0-128-9 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 17:14:48.685269 ip-10-0-128-9 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 18 17:14:48.685269 ip-10-0-128-9 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 18 17:14:48.686282 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.685371 2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 18 17:14:48.688448 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688424 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 17:14:48.688448 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688442 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 17:14:48.688448 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688447 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 17:14:48.688448 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688452 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688456 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688461 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688464 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688468 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688472 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688475 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688479 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688483 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688486 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688490 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688495 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688499 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688503 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688507 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688510 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688514 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688518 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688522 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688526 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 17:14:48.688693 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688540 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688544 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688548 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688552 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688557 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688561 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688565 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688569 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688573 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688577 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688584 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688590 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688596 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688601 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688609 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688614 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688620 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688625 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688629 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 17:14:48.689532 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688634 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688640 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688645 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688649 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688656 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688663 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688668 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688673 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688677 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688681 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688686 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688690 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688695 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688699 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688703 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688707 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688712 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688716 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688721 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688725 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 17:14:48.690351 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688729 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688733 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688738 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688742 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688746 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688750 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688755 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688760 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688765 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688770 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688774 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688779 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688783 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688789 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688793 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688797 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688802 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688806 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688810 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 17:14:48.691189 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688815 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688819 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688825 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688829 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688833 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688836 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688841 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688845 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688850 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688854 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688858 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688862 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688867 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688872 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688876 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688880 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688884 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.688889 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690064 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690077 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 17:14:48.692088 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690083 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690089 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690094 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690099 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690104 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690108 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690112 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690118 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690122 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690127 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690131 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690137 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690143 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690148 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690152 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690156 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690160 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690165 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690169 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690173 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 17:14:48.692976 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690178 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690182 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690186 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690190 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690195 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690202 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690206 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690211 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690215 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690219 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690223 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690229 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690233 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690237 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690241 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690245 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690250 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690255 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690259 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690263 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 17:14:48.693684 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690268 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690272 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690276 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690280 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690284 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690289 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690317 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690322 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690326 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690330 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690334 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690338 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690342 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690346 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690350 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690353 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690358 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690361 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690365 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690370 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 17:14:48.694311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690374 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690378 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690382 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690387 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690391 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690395 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690400 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690404 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690408 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690412 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690416 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690421 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690425 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690429 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690434 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690439 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690445 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690449 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690453 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690457 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 17:14:48.694842 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690461 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690466 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690470 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690475 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690479 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690483 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690488 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690492 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690497 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690501 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690505 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690509 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690515 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690520 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690525 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690530 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.690535 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691160 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691168 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 17:14:48.695431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691173 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691178 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691183 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691187 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691192 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691196 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691202 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691207 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691212 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691216 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691220 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691224 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691229 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691233 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691237 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691242 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691246 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691250 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691254 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691258 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 17:14:48.696072 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691262 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691267 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691271 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691276 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691280 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691284 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691288 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691314 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691318 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691323 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691327 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691332 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691336 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691340 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691345 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691349 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691353 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691357 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691361 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 17:14:48.696629 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691366 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691371 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691376 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691380 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691384 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691388 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691392 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691399 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691404 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691408 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691412 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691416 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691422 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691429 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691433 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691437 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691442 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691446 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691450 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 17:14:48.697184 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691454 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691458 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691463 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691467 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691471 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691476 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691479 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691483 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691489 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691494 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691498 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691502 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691506 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691513 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691519 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691523 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691528 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691532 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691536 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 18 17:14:48.697659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691540 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691544 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691548 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691554 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691558 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691562 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691566 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691571 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691575 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691580 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691586 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691592 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691597 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691602 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691607 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691612 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691617 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691622 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691627 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.691631 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 17:14:48.698150 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691738 2560 flags.go:64] FLAG: --address="0.0.0.0" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691785 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691796 2560 flags.go:64] FLAG: --anonymous-auth="true" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691803 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691809 2560 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691814 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691821 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691827 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691833 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691838 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691843 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691849 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691854 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691859 2560 flags.go:64] FLAG: --cgroup-root="" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691864 2560 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691869 2560 flags.go:64] FLAG: --client-ca-file="" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691873 2560 flags.go:64] FLAG: --cloud-provider="external" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691878 2560 flags.go:64] FLAG: --cluster-dns="[]" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691884 2560 flags.go:64] FLAG: --cluster-domain="" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691890 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691895 2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691900 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691905 2560 flags.go:64] FLAG: --container-log-max-files="5" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691911 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 18 17:14:48.698652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691916 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691921 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691926 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691931 2560 flags.go:64] FLAG: --contention-profiling="false" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691936 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691941 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691946 2560 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691951 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691957 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691962 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691967 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691971 2560 flags.go:64] FLAG: --enable-load-reader="false" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691976 2560 flags.go:64] FLAG: --enable-server="true" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691981 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691989 2560 flags.go:64] FLAG: --event-burst="100" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691994 2560 flags.go:64] FLAG: --event-qps="50" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.691999 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692004 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692009 2560 flags.go:64] FLAG: --eviction-hard="" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692016 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692020 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692025 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692030 2560 flags.go:64] FLAG: --eviction-soft="" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692035 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692039 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 18 17:14:48.699221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692044 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692049 2560 flags.go:64] FLAG: --experimental-mounter-path="" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692053 2560 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692058 2560 flags.go:64] FLAG: --fail-swap-on="true" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692063 2560 flags.go:64] FLAG: --feature-gates="" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692069 2560 flags.go:64] FLAG: --file-check-frequency="20s" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692074 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692079 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692084 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692089 2560 flags.go:64] FLAG: --healthz-port="10248" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692093 2560 flags.go:64] FLAG: --help="false" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692097 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-128-9.ec2.internal" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692102 2560 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692106 2560 flags.go:64] FLAG: --http-check-frequency="20s" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692111 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692116 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692122 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692127 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692136 2560 flags.go:64] FLAG: --image-service-endpoint="" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692141 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692145 2560 flags.go:64] FLAG: --kube-api-burst="100" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692150 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692155 2560 flags.go:64] FLAG: --kube-api-qps="50" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692159 2560 flags.go:64] FLAG: --kube-reserved="" Sep 18 17:14:48.699899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692165 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692170 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692175 2560 flags.go:64] FLAG: --kubelet-cgroups="" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692180 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692184 2560 flags.go:64] FLAG: --lock-file="" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692189 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692193 2560 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692198 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692209 2560 flags.go:64] FLAG: --log-json-split-stream="false" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692213 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692218 2560 flags.go:64] FLAG: --log-text-split-stream="false" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692223 2560 flags.go:64] FLAG: --logging-format="text" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692227 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692232 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692237 2560 flags.go:64] FLAG: --manifest-url="" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692242 2560 flags.go:64] FLAG: --manifest-url-header="" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692249 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692254 2560 flags.go:64] FLAG: --max-open-files="1000000" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692260 2560 flags.go:64] FLAG: --max-pods="110" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692265 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692270 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692274 2560 flags.go:64] FLAG: --memory-manager-policy="None" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692279 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692284 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692305 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 18 17:14:48.700554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692311 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692324 2560 flags.go:64] FLAG: --node-status-max-images="50" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692329 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692334 2560 flags.go:64] FLAG: --oom-score-adj="-999" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692338 2560 flags.go:64] FLAG: --pod-cidr="" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692343 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692351 2560 flags.go:64] FLAG: --pod-manifest-path="" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692356 2560 flags.go:64] FLAG: --pod-max-pids="-1" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692360 2560 flags.go:64] FLAG: --pods-per-core="0" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692366 2560 flags.go:64] FLAG: --port="10250" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692371 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692376 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02060cd6653d21970" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692381 2560 flags.go:64] FLAG: --qos-reserved="" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692385 2560 flags.go:64] FLAG: --read-only-port="10255" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692390 2560 flags.go:64] FLAG: --register-node="true" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692395 2560 flags.go:64] FLAG: --register-with-taints="" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692403 2560 flags.go:64] FLAG: --registry-burst="10" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692407 2560 flags.go:64] FLAG: --registry-qps="5" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692412 2560 flags.go:64] FLAG: --reserved-cpus="" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692416 2560 flags.go:64] FLAG: --reserved-memory="" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692422 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692427 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692432 2560 flags.go:64] FLAG: --rotate-certificates="false" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692437 2560 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692442 2560 flags.go:64] FLAG: --runonce="false" Sep 18 17:14:48.701164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692447 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692452 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692457 2560 flags.go:64] FLAG: --seccomp-default="false" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692461 2560 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692466 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692471 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692476 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692481 2560 flags.go:64] FLAG: --storage-driver-password="root" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692487 2560 flags.go:64] FLAG: --storage-driver-secure="false" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692491 2560 flags.go:64] FLAG: --storage-driver-table="stats" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692496 2560 flags.go:64] FLAG: --storage-driver-user="root" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692501 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692506 2560 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692511 2560 flags.go:64] FLAG: --system-cgroups="" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692516 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692524 2560 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692529 2560 flags.go:64] FLAG: --tls-cert-file="" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692535 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692541 2560 flags.go:64] FLAG: --tls-min-version="" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692545 2560 flags.go:64] FLAG: --tls-private-key-file="" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692550 2560 flags.go:64] FLAG: --topology-manager-policy="none" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692554 2560 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692559 2560 flags.go:64] FLAG: --topology-manager-scope="container" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692566 2560 flags.go:64] FLAG: --v="2" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692572 2560 flags.go:64] FLAG: --version="false" Sep 18 17:14:48.701852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692578 2560 flags.go:64] FLAG: --vmodule="" Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692584 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.692589 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692773 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692781 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692785 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692790 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692794 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692799 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692803 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692807 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692811 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692816 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692820 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692824 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692830 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692835 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692839 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692843 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 17:14:48.702488 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692848 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692852 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692856 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692860 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692864 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692869 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692884 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692889 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692893 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692897 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692902 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692908 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692912 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692916 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692922 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692928 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692933 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692937 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692942 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692947 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 17:14:48.702956 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692952 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692956 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692960 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692964 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692969 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692972 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692977 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692981 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692987 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692992 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.692999 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693005 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693009 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693013 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693018 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693022 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693026 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693031 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693035 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693040 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 17:14:48.703452 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693044 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693049 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693053 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693059 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693065 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693071 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693076 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693081 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693086 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693090 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693094 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693098 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693103 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693107 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693111 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693115 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693119 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693124 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693128 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693140 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 17:14:48.703938 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693147 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693151 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693155 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693160 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693164 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693168 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693172 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693176 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693181 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693185 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693189 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693193 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693198 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693203 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693207 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693214 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693218 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693222 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693226 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693230 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 17:14:48.704457 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693234 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693239 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.693243 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.693786 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.701476 2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.701490 2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.701508 2560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.701517 2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701570 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701575 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701578 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701581 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701585 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701588 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701591 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 17:14:48.704979 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701595 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701597 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701600 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701603 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701606 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701608 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701611 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701614 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701617 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701621 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701623 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701626 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701629 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701631 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701634 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701637 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701639 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701642 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701645 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 17:14:48.705445 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701647 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701650 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701652 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701658 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701662 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701665 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701668 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701671 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701674 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701677 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701680 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701683 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701685 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701688 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701691 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701694 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701697 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701700 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701703 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 17:14:48.705906 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701707 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701709 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701712 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701715 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701718 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701721 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701724 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701727 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701729 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701732 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701734 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701737 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701740 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701743 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701746 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701749 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701752 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701755 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701758 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701761 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 17:14:48.706397 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701763 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701765 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701768 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701771 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701773 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701776 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701778 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701781 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701783 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701786 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701789 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701791 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701794 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701797 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701800 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701802 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701805 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701808 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701811 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701813 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 18 17:14:48.706881 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701817 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701820 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701823 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701826 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701829 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701832 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701835 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701838 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701841 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701844 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701847 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701850 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701853 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701856 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.701862 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701970 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 18 17:14:48.707388 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701975 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701979 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701982 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701985 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701988 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701991 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701994 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701996 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.701999 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702002 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702004 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702008 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702012 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702015 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702019 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702021 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702024 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702027 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702029 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702032 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 18 17:14:48.707777 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702034 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702037 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702040 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702043 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702046 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702050 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702053 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702056 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702060 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702063 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702066 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702087 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702091 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702094 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702097 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702100 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702103 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702106 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702109 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 18 17:14:48.708380 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702112 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702115 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702117 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702120 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702122 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702125 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702127 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702130 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702133 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702136 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702139 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702141 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702144 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702146 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702149 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702152 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702154 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702157 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702159 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702162 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 18 17:14:48.708879 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702164 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702167 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702169 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702172 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702175 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702177 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702180 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702183 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702185 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702188 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702190 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702193 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702196 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702198 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702201 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702203 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702206 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702208 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702211 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702213 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 18 17:14:48.709644 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702216 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702220 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702222 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702225 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702227 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702230 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702232 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702235 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702238 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702241 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702244 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702246 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702249 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702251 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702254 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702256 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702259 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702261 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 18 17:14:48.710177 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:48.702264 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 18 17:14:48.710689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.702268 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 18 17:14:48.710689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.702377 2560 server.go:962] "Client rotation is on, will bootstrap in background" Sep 18 17:14:48.710689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.704334 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 18 17:14:48.710689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.705214 2560 server.go:1019] "Starting client certificate rotation" Sep 18 17:14:48.710689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.705324 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 17:14:48.710689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.705359 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 18 17:14:48.710689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.708127 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 17:14:48.711834 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.711818 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 18 17:14:48.718614 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.718591 2560 log.go:25] "Validated CRI v1 runtime API" Sep 18 17:14:48.725462 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.725446 2560 log.go:25] "Validated CRI v1 image API" Sep 18 17:14:48.726326 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.726309 2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 18 17:14:48.728495 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.728472 2560 fs.go:135] Filesystem UUIDs: map[646f588b-6160-4645-be1f-eb45d90895bc:/dev/nvme0n1p4 768f3538-f7e2-4c27-868c-474161e835ef:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 18 17:14:48.728552 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.728495 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 18 17:14:48.732379 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.732253 2560 manager.go:216] Machine: {Timestamp:2026-09-18 17:14:48.731780374 +0000 UTC m=+1.984941202 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096838 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27151205b4958553c6a27671b88474 SystemUUID:ec271512-05b4-9585-53c6-a27671b88474 BootID:2504eba0-b48b-487c-9228-7ef935e9d9ef Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e9:c7:87:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e9:c7:87:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:57:50:e4:49:f9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 18 17:14:48.732379 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.732379 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 18 17:14:48.732504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.732459 2560 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 18 17:14:48.734047 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.734026 2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 18 17:14:48.734179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.734049 2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-9.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 18 17:14:48.735160 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.735149 2560 topology_manager.go:138] "Creating topology manager with none policy" Sep 18 17:14:48.735197 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.735161 2560 container_manager_linux.go:309] "Creating device plugin manager" Sep 18 17:14:48.735197 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.735169 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 17:14:48.735197 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.735186 2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 18 17:14:48.735287 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.735252 2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 18 17:14:48.738005 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.737994 2560 state_mem.go:36] "Initialized new in-memory state store" Sep 18 17:14:48.738110 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.738101 2560 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 18 17:14:48.738139 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.738133 2560 kubelet.go:486] "Attempting to sync node with API server" Sep 18 17:14:48.738166 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.738141 2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 18 17:14:48.738166 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.738155 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 18 17:14:48.738166 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.738166 2560 kubelet.go:398] "Adding apiserver pod source" Sep 18 17:14:48.738250 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.738175 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 18 17:14:48.738627 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.738614 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 17:14:48.738671 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.738635 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 18 17:14:48.740753 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.740733 2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 18 17:14:48.743256 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.743243 2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 18 17:14:48.743336 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.743273 2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 18 17:14:48.744379 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744362 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 18 17:14:48.744379 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744379 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 18 17:14:48.744379 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744385 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 18 17:14:48.744504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744391 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 18 17:14:48.744504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744400 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 18 17:14:48.744504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744405 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 18 17:14:48.744504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744418 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 18 17:14:48.744504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744423 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 18 17:14:48.744504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744436 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 18 17:14:48.744504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744441 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 18 17:14:48.744504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744453 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 18 17:14:48.744504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744459 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 18 17:14:48.744504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744482 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 18 17:14:48.744504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.744487 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 18 17:14:48.750362 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.750326 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 17:14:48.752518 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.752494 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 18 17:14:48.752611 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.752521 2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-9.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 18 17:14:48.753010 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.752985 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 18 17:14:48.753752 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.753739 2560 server.go:1268] "Started kubelet" Sep 18 17:14:48.753820 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.753797 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 18 17:14:48.753931 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.753882 2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 18 17:14:48.754020 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.753947 2560 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 18 17:14:48.754553 ip-10-0-128-9 systemd[1]: Started Kubernetes Kubelet. Sep 18 17:14:48.755017 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.754963 2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 18 17:14:48.755117 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.755097 2560 server.go:310] "Adding debug handlers to kubelet server" Sep 18 17:14:48.759027 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.759007 2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 18 17:14:48.760493 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.760479 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 18 17:14:48.760564 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.760503 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 18 17:14:48.760564 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.760549 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:48.761182 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.761165 2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 18 17:14:48.761257 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.761193 2560 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 18 17:14:48.761257 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.761208 2560 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 18 17:14:48.761477 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.761458 2560 factory.go:153] Registering CRI-O factory Sep 18 17:14:48.761477 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.761477 2560 factory.go:223] Registration of the crio container factory successfully Sep 18 17:14:48.761619 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.761520 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 18 17:14:48.761619 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.761529 2560 factory.go:55] Registering systemd factory Sep 18 17:14:48.761619 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.761606 2560 factory.go:223] Registration of the systemd container factory successfully Sep 18 17:14:48.761718 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.761627 2560 factory.go:103] Registering Raw factory Sep 18 17:14:48.761718 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.761650 2560 manager.go:1196] Started watching for new ooms in manager Sep 18 17:14:48.762084 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.762071 2560 reconstruct.go:97] "Volume reconstruction finished" Sep 18 17:14:48.762084 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.762082 2560 reconciler.go:29] "Reconciler: start to sync state" Sep 18 17:14:48.762181 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.762073 2560 manager.go:349] Starting recovery of all containers Sep 18 17:14:48.763560 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.763534 2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 18 17:14:48.763642 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.763627 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 18 17:14:48.771827 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.771811 2560 manager.go:354] Recovery completed Sep 18 17:14:48.776273 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.776262 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:14:48.791044 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.791020 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7fdls" Sep 18 17:14:48.791576 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.791538 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:14:48.791676 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.791594 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:14:48.791676 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.791610 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:14:48.792458 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.792439 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 18 17:14:48.792458 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.792456 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 18 17:14:48.792585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.792474 2560 state_mem.go:36] "Initialized new in-memory state store" Sep 18 17:14:48.792585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.792444 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:14:48.793354 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.793340 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:14:48.793432 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.793365 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:14:48.793432 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.793376 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:14:48.793559 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.793545 2560 policy_none.go:49] "None policy: Start" Sep 18 17:14:48.793608 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.793566 2560 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 18 17:14:48.793608 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.793581 2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 18 17:14:48.794605 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.794590 2560 policy_none.go:47] "Start" Sep 18 17:14:48.802408 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.802390 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7fdls" Sep 18 17:14:48.806480 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.805525 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-9.ec2.internal.18d679adac51f420 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-9.ec2.internal,UID:ip-10-0-128-9.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-9.ec2.internal,},FirstTimestamp:2026-09-18 17:14:48.753706016 +0000 UTC m=+2.006866826,LastTimestamp:2026-09-18 17:14:48.753706016 +0000 UTC m=+2.006866826,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-9.ec2.internal,}" Sep 18 17:14:48.839888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.826819 2560 manager.go:337] "Starting Device Plugin manager" Sep 18 17:14:48.839888 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.826858 2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 18 17:14:48.839888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.826868 2560 server.go:87] "Starting device plugin registration server" Sep 18 17:14:48.839888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.827081 2560 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 18 17:14:48.839888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.827098 2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 18 17:14:48.839888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.827187 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 18 17:14:48.839888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.827279 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 18 17:14:48.839888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.827288 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 18 17:14:48.839888 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.827897 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 18 17:14:48.839888 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.827950 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:48.856381 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.856354 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 18 17:14:48.857542 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.857526 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 18 17:14:48.857675 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.857581 2560 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 18 17:14:48.857675 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.857608 2560 kubelet.go:2448] "Starting kubelet main sync loop" Sep 18 17:14:48.857675 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.857653 2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 18 17:14:48.859730 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.859709 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 17:14:48.927923 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.927903 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:14:48.928792 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.928779 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:14:48.928869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.928805 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:14:48.928869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.928816 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:14:48.928869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.928841 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-9.ec2.internal" Sep 18 17:14:48.936626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.936568 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-9.ec2.internal" Sep 18 17:14:48.936626 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.936592 2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-9.ec2.internal\": node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:48.956379 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.956363 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:48.958413 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.958390 2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-9.ec2.internal"] Sep 18 17:14:48.958486 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.958470 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:14:48.959263 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.959249 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:14:48.959349 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.959277 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:14:48.959349 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.959309 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:14:48.961381 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.961367 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:14:48.961492 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.961477 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal" Sep 18 17:14:48.961542 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.961504 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:14:48.962054 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.962036 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:14:48.962054 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.962049 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:14:48.962185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.962063 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:14:48.962185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.962068 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:14:48.962185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.962073 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:14:48.962185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.962078 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:14:48.962827 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.962810 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eccee8496a5198ea9303df36f6adb7ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal\" (UID: \"eccee8496a5198ea9303df36f6adb7ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal" Sep 18 17:14:48.962903 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.962849 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eccee8496a5198ea9303df36f6adb7ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal\" (UID: \"eccee8496a5198ea9303df36f6adb7ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal" Sep 18 17:14:48.962903 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.962893 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e55b801a498c78c600fd5d152605bab9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-9.ec2.internal\" (UID: \"e55b801a498c78c600fd5d152605bab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-9.ec2.internal" Sep 18 17:14:48.964197 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.964181 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-9.ec2.internal" Sep 18 17:14:48.964248 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.964213 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:14:48.964898 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.964885 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:14:48.964956 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.964909 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:14:48.964956 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:48.964922 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:14:48.994862 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.994847 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-9.ec2.internal\" not found" node="ip-10-0-128-9.ec2.internal" Sep 18 17:14:48.999033 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:48.999019 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-9.ec2.internal\" not found" node="ip-10-0-128-9.ec2.internal" Sep 18 17:14:49.057472 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:49.057451 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:49.063776 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.063760 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eccee8496a5198ea9303df36f6adb7ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal\" (UID: \"eccee8496a5198ea9303df36f6adb7ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal" Sep 18 17:14:49.063861 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.063784 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eccee8496a5198ea9303df36f6adb7ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal\" (UID: \"eccee8496a5198ea9303df36f6adb7ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal" Sep 18 17:14:49.063861 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.063802 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e55b801a498c78c600fd5d152605bab9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-9.ec2.internal\" (UID: \"e55b801a498c78c600fd5d152605bab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-9.ec2.internal" Sep 18 17:14:49.063861 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.063840 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e55b801a498c78c600fd5d152605bab9-config\") pod \"kube-apiserver-proxy-ip-10-0-128-9.ec2.internal\" (UID: \"e55b801a498c78c600fd5d152605bab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-9.ec2.internal" Sep 18 17:14:49.063981 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.063873 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eccee8496a5198ea9303df36f6adb7ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal\" (UID: \"eccee8496a5198ea9303df36f6adb7ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal" Sep 18 17:14:49.063981 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.063876 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eccee8496a5198ea9303df36f6adb7ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal\" (UID: \"eccee8496a5198ea9303df36f6adb7ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal" Sep 18 17:14:49.158453 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:49.158428 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:49.259204 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:49.259137 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:49.298597 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.298571 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal" Sep 18 17:14:49.302325 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.302306 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-9.ec2.internal" Sep 18 17:14:49.359331 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:49.359282 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:49.459773 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:49.459747 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:49.560305 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:49.560233 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:49.660689 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:49.660658 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:49.693657 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.693631 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 17:14:49.705415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.705392 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 18 17:14:49.705558 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.705540 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 18 17:14:49.705558 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.705548 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 18 17:14:49.761118 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:49.761087 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:49.807976 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.807938 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 17:09:48 +0000 UTC" deadline="2028-05-22 23:26:15.916103749 +0000 UTC" Sep 18 17:14:49.807976 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.807970 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14694h11m26.108137395s" Sep 18 17:14:49.861596 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:49.861531 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:49.924624 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:49.924593 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccee8496a5198ea9303df36f6adb7ef.slice/crio-730560503ad9057d979f7d68615974a784e29fac2107e35bb7f00d207e7270dd WatchSource:0}: Error finding container 730560503ad9057d979f7d68615974a784e29fac2107e35bb7f00d207e7270dd: Status 404 returned error can't find the container with id 730560503ad9057d979f7d68615974a784e29fac2107e35bb7f00d207e7270dd Sep 18 17:14:49.924962 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:49.924942 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55b801a498c78c600fd5d152605bab9.slice/crio-abb35ed37e405ef2772debba127fce04a4b7e7dcd5f3e247016c5d3f3cfaebf6 WatchSource:0}: Error finding container abb35ed37e405ef2772debba127fce04a4b7e7dcd5f3e247016c5d3f3cfaebf6: Status 404 returned error can't find the container with id abb35ed37e405ef2772debba127fce04a4b7e7dcd5f3e247016c5d3f3cfaebf6 Sep 18 17:14:49.929732 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.929718 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 17:14:49.937423 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.937405 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 18 17:14:49.938138 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.938121 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientMemory" Sep 18 17:14:49.938217 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.938153 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 18 17:14:49.938217 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:49.938168 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeHasSufficientPID" Sep 18 17:14:49.962287 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:49.962265 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:50.062729 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:50.062701 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:50.163253 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:50.163194 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-9.ec2.internal\" not found" Sep 18 17:14:50.196700 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.195933 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 17:14:50.261637 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.261614 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal" Sep 18 17:14:50.272658 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.272640 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 17:14:50.273586 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.273571 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-9.ec2.internal" Sep 18 17:14:50.281369 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.281354 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 18 17:14:50.361088 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.361066 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 17:14:50.739459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.739429 2560 apiserver.go:52] "Watching apiserver" Sep 18 17:14:50.742188 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.742158 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 17:14:50.746878 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.746854 2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 18 17:14:50.747320 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.747281 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-qrbrr","kube-system/konnectivity-agent-78r8w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal","openshift-multus/multus-additional-cni-plugins-x7ns6","openshift-multus/multus-s2ln6","openshift-multus/network-metrics-daemon-clj2d","openshift-network-diagnostics/network-check-target-wqx8x","openshift-network-operator/iptables-alerter-5fdzk","kube-system/kube-apiserver-proxy-ip-10-0-128-9.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5","openshift-cluster-node-tuning-operator/tuned-jtcp7","openshift-dns/node-resolver-4vxw9","openshift-image-registry/node-ca-44qw2","openshift-ovn-kubernetes/ovnkube-node-qf4rc"] Sep 18 17:14:50.749940 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.749918 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.752143 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.752121 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-44qw2" Sep 18 17:14:50.752894 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.752873 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z25s6\"" Sep 18 17:14:50.752983 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.752913 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 18 17:14:50.753045 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.752995 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 18 17:14:50.753097 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.753062 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 18 17:14:50.754531 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.754510 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:50.754617 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:50.754590 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:14:50.754676 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.754641 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 18 17:14:50.754728 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.754688 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 18 17:14:50.754829 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.754813 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 18 17:14:50.754918 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.754898 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5rn2p\"" Sep 18 17:14:50.756905 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.756882 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.759195 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.759182 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.759435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.759414 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-54h9t\"" Sep 18 17:14:50.759604 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.759591 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 18 17:14:50.759671 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.759613 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 18 17:14:50.759741 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.759726 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 18 17:14:50.760100 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.759731 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 18 17:14:50.760100 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.760044 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 18 17:14:50.761554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.761372 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 18 17:14:50.761807 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.761790 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 18 17:14:50.761885 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.761841 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5lxbz\"" Sep 18 17:14:50.762193 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.762016 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:14:50.762193 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:50.762085 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:14:50.764878 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.763857 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-78r8w" Sep 18 17:14:50.766243 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.766224 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 18 17:14:50.766243 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.766239 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c4zw5\"" Sep 18 17:14:50.766398 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.766249 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 18 17:14:50.766600 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.766584 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:50.766682 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:50.766664 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:14:50.769277 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.769233 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.771696 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.771676 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 18 17:14:50.771797 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.771705 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-var-lib-kubelet\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.771797 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.771748 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-sys-fs\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.771797 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.771779 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73-host\") pod \"node-ca-44qw2\" (UID: \"b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73\") " pod="openshift-image-registry/node-ca-44qw2" Sep 18 17:14:50.771955 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.771815 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hnnl\" (UniqueName: \"kubernetes.io/projected/492a5d0b-c698-4eab-b0d6-5c2961725674-kube-api-access-5hnnl\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.771955 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.771852 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:50.771955 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.771879 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4ft2\"" Sep 18 17:14:50.772097 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.771883 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-etc-selinux\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.772097 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.772076 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljkjc\" (UniqueName: \"kubernetes.io/projected/6cb69917-471e-4d0a-bba9-60d8552d49fb-kube-api-access-ljkjc\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.772191 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.772105 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03103480-84b5-46bf-a5ac-03f8aac2216a-cni-binary-copy\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.772191 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.772138 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-device-dir\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.772191 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.772148 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 18 17:14:50.772888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.772692 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-system-cni-dir\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.772888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.772816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-hostroot\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.773030 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.772932 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-cnibin\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.773030 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.772982 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-registration-dir\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.773125 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773031 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/492a5d0b-c698-4eab-b0d6-5c2961725674-cnibin\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.773125 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773067 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/492a5d0b-c698-4eab-b0d6-5c2961725674-cni-binary-copy\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.773125 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773098 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/492a5d0b-c698-4eab-b0d6-5c2961725674-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.773274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773153 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/492a5d0b-c698-4eab-b0d6-5c2961725674-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.773274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773223 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-etc-kubernetes\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.773274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773265 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkzxm\" (UniqueName: \"kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm\") pod \"network-check-target-wqx8x\" (UID: \"6883d049-759d-4fc5-af31-d30305f4b092\") " pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:14:50.773433 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773309 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.773433 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773372 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4vxw9" Sep 18 17:14:50.773433 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773375 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-socket-dir\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.773572 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773438 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/492a5d0b-c698-4eab-b0d6-5c2961725674-system-cni-dir\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.773572 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773475 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-multus-cni-dir\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.773572 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773533 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-run-netns\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.773705 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773567 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4tpz\" (UniqueName: \"kubernetes.io/projected/03103480-84b5-46bf-a5ac-03f8aac2216a-kube-api-access-z4tpz\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.773705 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773643 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhx69\" (UniqueName: \"kubernetes.io/projected/cd4d7977-637e-4068-8d44-515534534834-kube-api-access-rhx69\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:50.773801 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773681 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t668c\" (UniqueName: \"kubernetes.io/projected/b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73-kube-api-access-t668c\") pod \"node-ca-44qw2\" (UID: \"b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73\") " pod="openshift-image-registry/node-ca-44qw2" Sep 18 17:14:50.773801 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773749 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/492a5d0b-c698-4eab-b0d6-5c2961725674-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.773887 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773800 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-multus-conf-dir\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.773958 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773937 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/03103480-84b5-46bf-a5ac-03f8aac2216a-multus-daemon-config\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.774063 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.773972 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-run-multus-certs\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.774063 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.774005 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/156fc6c4-1430-428e-aa27-fb144f9ab893-agent-certs\") pod \"konnectivity-agent-78r8w\" (UID: \"156fc6c4-1430-428e-aa27-fb144f9ab893\") " pod="kube-system/konnectivity-agent-78r8w" Sep 18 17:14:50.774063 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.774056 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-run-k8s-cni-cncf-io\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.774203 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.774093 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-var-lib-cni-bin\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.774203 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.774150 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-var-lib-cni-multus\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.774503 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.774208 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/156fc6c4-1430-428e-aa27-fb144f9ab893-konnectivity-ca\") pod \"konnectivity-agent-78r8w\" (UID: \"156fc6c4-1430-428e-aa27-fb144f9ab893\") " pod="kube-system/konnectivity-agent-78r8w" Sep 18 17:14:50.774503 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.774247 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73-serviceca\") pod \"node-ca-44qw2\" (UID: \"b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73\") " pod="openshift-image-registry/node-ca-44qw2" Sep 18 17:14:50.774503 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.774276 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-multus-socket-dir-parent\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.775797 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.775740 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 18 17:14:50.776073 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.776045 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 18 17:14:50.776578 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.776325 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 18 17:14:50.776578 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.776395 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ptg44\"" Sep 18 17:14:50.776862 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.776837 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5fdzk" Sep 18 17:14:50.779121 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.779103 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 18 17:14:50.779858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.779460 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 18 17:14:50.779858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.779575 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.779858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.779672 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sqknz\"" Sep 18 17:14:50.779858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.779758 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 18 17:14:50.781794 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.781775 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m626r\"" Sep 18 17:14:50.782238 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.782220 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 18 17:14:50.782238 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.782234 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 18 17:14:50.782387 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.782269 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 18 17:14:50.782387 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.782353 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 18 17:14:50.782485 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.782408 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 18 17:14:50.782534 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.782492 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 18 17:14:50.813475 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.813452 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bxvqd" Sep 18 17:14:50.822263 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.822241 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bxvqd" Sep 18 17:14:50.862077 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.862052 2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 18 17:14:50.862771 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.862723 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal" event={"ID":"eccee8496a5198ea9303df36f6adb7ef","Type":"ContainerStarted","Data":"730560503ad9057d979f7d68615974a784e29fac2107e35bb7f00d207e7270dd"} Sep 18 17:14:50.863994 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.863971 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-9.ec2.internal" event={"ID":"e55b801a498c78c600fd5d152605bab9","Type":"ContainerStarted","Data":"abb35ed37e405ef2772debba127fce04a4b7e7dcd5f3e247016c5d3f3cfaebf6"} Sep 18 17:14:50.874811 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.874788 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzxm\" (UniqueName: \"kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm\") pod \"network-check-target-wqx8x\" (UID: \"6883d049-759d-4fc5-af31-d30305f4b092\") " pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:14:50.874947 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.874815 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.874947 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.874853 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-multus-cni-dir\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.874947 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.874874 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17ee44c6-7ddd-473c-8b5b-c2e908141e09-ovn-node-metrics-cert\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.874947 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.874910 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-sysctl-conf\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.874947 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.874934 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t668c\" (UniqueName: \"kubernetes.io/projected/b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73-kube-api-access-t668c\") pod \"node-ca-44qw2\" (UID: \"b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73\") " pod="openshift-image-registry/node-ca-44qw2" Sep 18 17:14:50.874947 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.874948 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-multus-conf-dir\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.874955 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.875232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.874964 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/03103480-84b5-46bf-a5ac-03f8aac2216a-multus-daemon-config\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875018 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-run-multus-certs\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875046 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-systemd-units\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.875232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875069 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-run-ovn\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.875232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875087 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-systemd\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.875232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875137 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-run-multus-certs\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875193 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-multus-conf-dir\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875229 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-var-lib-kubelet\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875228 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-multus-cni-dir\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875253 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-var-lib-cni-bin\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875287 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-var-lib-cni-bin\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875313 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-var-lib-cni-multus\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875345 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-log-socket\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875369 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/156fc6c4-1430-428e-aa27-fb144f9ab893-konnectivity-ca\") pod \"konnectivity-agent-78r8w\" (UID: \"156fc6c4-1430-428e-aa27-fb144f9ab893\") " pod="kube-system/konnectivity-agent-78r8w" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875396 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73-serviceca\") pod \"node-ca-44qw2\" (UID: \"b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73\") " pod="openshift-image-registry/node-ca-44qw2" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875423 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-multus-socket-dir-parent\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875421 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-var-lib-cni-multus\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875446 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-var-lib-kubelet\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875470 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a44888c-a9bf-4378-8785-26bc77ef43bb-hosts-file\") pod \"node-resolver-4vxw9\" (UID: \"4a44888c-a9bf-4378-8785-26bc77ef43bb\") " pod="openshift-dns/node-resolver-4vxw9" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875504 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-multus-socket-dir-parent\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875510 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a44888c-a9bf-4378-8785-26bc77ef43bb-tmp-dir\") pod \"node-resolver-4vxw9\" (UID: \"4a44888c-a9bf-4378-8785-26bc77ef43bb\") " pod="openshift-dns/node-resolver-4vxw9" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875534 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73-host\") pod \"node-ca-44qw2\" (UID: \"b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73\") " pod="openshift-image-registry/node-ca-44qw2" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875560 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-var-lib-kubelet\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.875662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875560 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hnnl\" (UniqueName: \"kubernetes.io/projected/492a5d0b-c698-4eab-b0d6-5c2961725674-kube-api-access-5hnnl\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875733 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-run-openvswitch\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875773 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73-host\") pod \"node-ca-44qw2\" (UID: \"b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73\") " pod="openshift-image-registry/node-ca-44qw2" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875827 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-node-log\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875869 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875879 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73-serviceca\") pod \"node-ca-44qw2\" (UID: \"b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73\") " pod="openshift-image-registry/node-ca-44qw2" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875892 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-etc-selinux\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875919 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljkjc\" (UniqueName: \"kubernetes.io/projected/6cb69917-471e-4d0a-bba9-60d8552d49fb-kube-api-access-ljkjc\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875958 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/156fc6c4-1430-428e-aa27-fb144f9ab893-konnectivity-ca\") pod \"konnectivity-agent-78r8w\" (UID: \"156fc6c4-1430-428e-aa27-fb144f9ab893\") " pod="kube-system/konnectivity-agent-78r8w" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:50.875971 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.875960 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03103480-84b5-46bf-a5ac-03f8aac2216a-cni-binary-copy\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876006 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/af831002-a9cc-4239-9821-98618e18e764-dbus\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876022 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-etc-selinux\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876031 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/03103480-84b5-46bf-a5ac-03f8aac2216a-multus-daemon-config\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:50.876045 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs podName:cd4d7977-637e-4068-8d44-515534534834 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:51.376024172 +0000 UTC m=+4.629184982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs") pod "network-metrics-daemon-clj2d" (UID: "cd4d7977-637e-4068-8d44-515534534834") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876064 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-device-dir\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876106 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-system-cni-dir\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.876415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876112 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-device-dir\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876132 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-var-lib-openvswitch\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876174 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876164 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-system-cni-dir\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876200 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-kubernetes\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876230 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-cnibin\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876255 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-registration-dir\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876270 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/492a5d0b-c698-4eab-b0d6-5c2961725674-cnibin\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876346 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/492a5d0b-c698-4eab-b0d6-5c2961725674-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876349 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-cnibin\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876367 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-registration-dir\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876390 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/492a5d0b-c698-4eab-b0d6-5c2961725674-cnibin\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876387 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17ee44c6-7ddd-473c-8b5b-c2e908141e09-env-overrides\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876425 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17ee44c6-7ddd-473c-8b5b-c2e908141e09-ovnkube-script-lib\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876437 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03103480-84b5-46bf-a5ac-03f8aac2216a-cni-binary-copy\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876452 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-socket-dir\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876491 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/492a5d0b-c698-4eab-b0d6-5c2961725674-system-cni-dir\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.877235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876508 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/492a5d0b-c698-4eab-b0d6-5c2961725674-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876532 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-run-netns\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876561 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-run-systemd\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876576 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-socket-dir\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876549 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/492a5d0b-c698-4eab-b0d6-5c2961725674-system-cni-dir\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876593 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-sysctl-d\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876625 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-run-netns\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876624 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdhrk\" (UniqueName: \"kubernetes.io/projected/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-kube-api-access-bdhrk\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876662 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhx69\" (UniqueName: \"kubernetes.io/projected/cd4d7977-637e-4068-8d44-515534534834-kube-api-access-rhx69\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876690 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/492a5d0b-c698-4eab-b0d6-5c2961725674-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876718 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-etc-openvswitch\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876745 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-cni-bin\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876768 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-run\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876792 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-host\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876815 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/156fc6c4-1430-428e-aa27-fb144f9ab893-agent-certs\") pod \"konnectivity-agent-78r8w\" (UID: \"156fc6c4-1430-428e-aa27-fb144f9ab893\") " pod="kube-system/konnectivity-agent-78r8w" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876839 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-run-k8s-cni-cncf-io\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876863 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/645e614d-3ae7-4057-9b97-e7386685af67-host-slash\") pod \"iptables-alerter-5fdzk\" (UID: \"645e614d-3ae7-4057-9b97-e7386685af67\") " pod="openshift-network-operator/iptables-alerter-5fdzk" Sep 18 17:14:50.877965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.876931 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-host-run-k8s-cni-cncf-io\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.877053 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-tmp\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.877086 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48mnv\" (UniqueName: \"kubernetes.io/projected/4a44888c-a9bf-4378-8785-26bc77ef43bb-kube-api-access-48mnv\") pod \"node-resolver-4vxw9\" (UID: \"4a44888c-a9bf-4378-8785-26bc77ef43bb\") " pod="openshift-dns/node-resolver-4vxw9" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.877175 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vgxz\" (UniqueName: \"kubernetes.io/projected/17ee44c6-7ddd-473c-8b5b-c2e908141e09-kube-api-access-4vgxz\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.877204 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-sys-fs\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.877234 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/645e614d-3ae7-4057-9b97-e7386685af67-iptables-alerter-script\") pod \"iptables-alerter-5fdzk\" (UID: \"645e614d-3ae7-4057-9b97-e7386685af67\") " pod="openshift-network-operator/iptables-alerter-5fdzk" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.877258 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmcc9\" (UniqueName: \"kubernetes.io/projected/645e614d-3ae7-4057-9b97-e7386685af67-kube-api-access-gmcc9\") pod \"iptables-alerter-5fdzk\" (UID: \"645e614d-3ae7-4057-9b97-e7386685af67\") " pod="openshift-network-operator/iptables-alerter-5fdzk" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.877261 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/492a5d0b-c698-4eab-b0d6-5c2961725674-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.877274 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-slash\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.877670 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.877780 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cb69917-471e-4d0a-bba9-60d8552d49fb-sys-fs\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.877325 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-run-ovn-kubernetes\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.877983 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-cni-netd\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878093 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-sysconfig\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878125 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-sys\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878165 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17ee44c6-7ddd-473c-8b5b-c2e908141e09-ovnkube-config\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878195 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-lib-modules\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.878734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878265 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-tuned\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878327 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-hostroot\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878367 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/af831002-a9cc-4239-9821-98618e18e764-kubelet-config\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878403 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-run-netns\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878436 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878465 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-kubelet\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878493 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/492a5d0b-c698-4eab-b0d6-5c2961725674-cni-binary-copy\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878527 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/492a5d0b-c698-4eab-b0d6-5c2961725674-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878558 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-etc-kubernetes\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878589 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4tpz\" (UniqueName: \"kubernetes.io/projected/03103480-84b5-46bf-a5ac-03f8aac2216a-kube-api-access-z4tpz\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878617 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-modprobe-d\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878784 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-hostroot\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.878961 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03103480-84b5-46bf-a5ac-03f8aac2216a-etc-kubernetes\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.879275 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/492a5d0b-c698-4eab-b0d6-5c2961725674-cni-binary-copy\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.879465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.879444 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/492a5d0b-c698-4eab-b0d6-5c2961725674-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.881947 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.881929 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/156fc6c4-1430-428e-aa27-fb144f9ab893-agent-certs\") pod \"konnectivity-agent-78r8w\" (UID: \"156fc6c4-1430-428e-aa27-fb144f9ab893\") " pod="kube-system/konnectivity-agent-78r8w" Sep 18 17:14:50.883409 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:50.883385 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:14:50.883409 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:50.883414 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:14:50.883593 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:50.883428 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dkzxm for pod openshift-network-diagnostics/network-check-target-wqx8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:14:50.883593 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:50.883483 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm podName:6883d049-759d-4fc5-af31-d30305f4b092 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:51.383466312 +0000 UTC m=+4.636627122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dkzxm" (UniqueName: "kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm") pod "network-check-target-wqx8x" (UID: "6883d049-759d-4fc5-af31-d30305f4b092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:14:50.886196 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.886171 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t668c\" (UniqueName: \"kubernetes.io/projected/b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73-kube-api-access-t668c\") pod \"node-ca-44qw2\" (UID: \"b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73\") " pod="openshift-image-registry/node-ca-44qw2" Sep 18 17:14:50.886333 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.886310 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hnnl\" (UniqueName: \"kubernetes.io/projected/492a5d0b-c698-4eab-b0d6-5c2961725674-kube-api-access-5hnnl\") pod \"multus-additional-cni-plugins-x7ns6\" (UID: \"492a5d0b-c698-4eab-b0d6-5c2961725674\") " pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:50.886395 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.886320 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljkjc\" (UniqueName: \"kubernetes.io/projected/6cb69917-471e-4d0a-bba9-60d8552d49fb-kube-api-access-ljkjc\") pod \"aws-ebs-csi-driver-node-klqz5\" (UID: \"6cb69917-471e-4d0a-bba9-60d8552d49fb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:50.886931 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.886909 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhx69\" (UniqueName: \"kubernetes.io/projected/cd4d7977-637e-4068-8d44-515534534834-kube-api-access-rhx69\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:50.887370 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.887354 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4tpz\" (UniqueName: \"kubernetes.io/projected/03103480-84b5-46bf-a5ac-03f8aac2216a-kube-api-access-z4tpz\") pod \"multus-s2ln6\" (UID: \"03103480-84b5-46bf-a5ac-03f8aac2216a\") " pod="openshift-multus/multus-s2ln6" Sep 18 17:14:50.979192 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979158 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/af831002-a9cc-4239-9821-98618e18e764-dbus\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:50.979192 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979196 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-var-lib-openvswitch\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979219 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979243 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-kubernetes\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979267 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17ee44c6-7ddd-473c-8b5b-c2e908141e09-env-overrides\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979303 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17ee44c6-7ddd-473c-8b5b-c2e908141e09-ovnkube-script-lib\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979305 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/af831002-a9cc-4239-9821-98618e18e764-dbus\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979311 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979289 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-var-lib-openvswitch\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979332 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-run-systemd\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979350 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-kubernetes\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979371 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-run-systemd\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979398 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-sysctl-d\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979422 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdhrk\" (UniqueName: \"kubernetes.io/projected/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-kube-api-access-bdhrk\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979444 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-etc-openvswitch\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.979459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979465 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-cni-bin\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979487 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-run\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979507 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-host\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979532 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/645e614d-3ae7-4057-9b97-e7386685af67-host-slash\") pod \"iptables-alerter-5fdzk\" (UID: \"645e614d-3ae7-4057-9b97-e7386685af67\") " pod="openshift-network-operator/iptables-alerter-5fdzk" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979576 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/645e614d-3ae7-4057-9b97-e7386685af67-host-slash\") pod \"iptables-alerter-5fdzk\" (UID: \"645e614d-3ae7-4057-9b97-e7386685af67\") " pod="openshift-network-operator/iptables-alerter-5fdzk" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979583 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-sysctl-d\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979577 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-cni-bin\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979628 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-host\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979626 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-run\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979640 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-etc-openvswitch\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979629 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-tmp\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979684 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48mnv\" (UniqueName: \"kubernetes.io/projected/4a44888c-a9bf-4378-8785-26bc77ef43bb-kube-api-access-48mnv\") pod \"node-resolver-4vxw9\" (UID: \"4a44888c-a9bf-4378-8785-26bc77ef43bb\") " pod="openshift-dns/node-resolver-4vxw9" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979710 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vgxz\" (UniqueName: \"kubernetes.io/projected/17ee44c6-7ddd-473c-8b5b-c2e908141e09-kube-api-access-4vgxz\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979739 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/645e614d-3ae7-4057-9b97-e7386685af67-iptables-alerter-script\") pod \"iptables-alerter-5fdzk\" (UID: \"645e614d-3ae7-4057-9b97-e7386685af67\") " pod="openshift-network-operator/iptables-alerter-5fdzk" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979763 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmcc9\" (UniqueName: \"kubernetes.io/projected/645e614d-3ae7-4057-9b97-e7386685af67-kube-api-access-gmcc9\") pod \"iptables-alerter-5fdzk\" (UID: \"645e614d-3ae7-4057-9b97-e7386685af67\") " pod="openshift-network-operator/iptables-alerter-5fdzk" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979786 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-slash\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979813 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-run-ovn-kubernetes\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979822 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17ee44c6-7ddd-473c-8b5b-c2e908141e09-env-overrides\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979837 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-cni-netd\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979860 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-sysconfig\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979885 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-sys\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979932 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17ee44c6-7ddd-473c-8b5b-c2e908141e09-ovnkube-config\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979957 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-lib-modules\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979958 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-slash\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979940 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-run-ovn-kubernetes\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.979982 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-tuned\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980016 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/af831002-a9cc-4239-9821-98618e18e764-kubelet-config\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980023 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-cni-netd\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980034 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-sysconfig\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980041 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-run-netns\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980071 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-sys\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980071 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980107 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-kubelet\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980108 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-lib-modules\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:50.980135 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980148 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-modprobe-d\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.980954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980156 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-kubelet\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:50.980184 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret podName:af831002-a9cc-4239-9821-98618e18e764 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:51.480170185 +0000 UTC m=+4.733330980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret") pod "global-pull-secret-syncer-qrbrr" (UID: "af831002-a9cc-4239-9821-98618e18e764") : object "kube-system"/"original-pull-secret" not registered Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980207 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/af831002-a9cc-4239-9821-98618e18e764-kubelet-config\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980221 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-host-run-netns\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980228 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-modprobe-d\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980276 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17ee44c6-7ddd-473c-8b5b-c2e908141e09-ovn-node-metrics-cert\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980322 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-sysctl-conf\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980354 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17ee44c6-7ddd-473c-8b5b-c2e908141e09-ovnkube-script-lib\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980350 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-systemd-units\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980393 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-run-ovn\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980407 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/645e614d-3ae7-4057-9b97-e7386685af67-iptables-alerter-script\") pod \"iptables-alerter-5fdzk\" (UID: \"645e614d-3ae7-4057-9b97-e7386685af67\") " pod="openshift-network-operator/iptables-alerter-5fdzk" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980415 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-systemd\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980420 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-systemd-units\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980438 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-var-lib-kubelet\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980458 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-run-ovn\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980464 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-log-socket\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980472 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-systemd\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.981901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980493 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a44888c-a9bf-4378-8785-26bc77ef43bb-hosts-file\") pod \"node-resolver-4vxw9\" (UID: \"4a44888c-a9bf-4378-8785-26bc77ef43bb\") " pod="openshift-dns/node-resolver-4vxw9" Sep 18 17:14:50.982726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980503 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-var-lib-kubelet\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.982726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980510 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-log-socket\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.982726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980511 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-sysctl-conf\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.982726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980520 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a44888c-a9bf-4378-8785-26bc77ef43bb-tmp-dir\") pod \"node-resolver-4vxw9\" (UID: \"4a44888c-a9bf-4378-8785-26bc77ef43bb\") " pod="openshift-dns/node-resolver-4vxw9" Sep 18 17:14:50.982726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980553 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4a44888c-a9bf-4378-8785-26bc77ef43bb-hosts-file\") pod \"node-resolver-4vxw9\" (UID: \"4a44888c-a9bf-4378-8785-26bc77ef43bb\") " pod="openshift-dns/node-resolver-4vxw9" Sep 18 17:14:50.982726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980557 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-run-openvswitch\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.982726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980582 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-node-log\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.982726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980639 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-run-openvswitch\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.982726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980658 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17ee44c6-7ddd-473c-8b5b-c2e908141e09-node-log\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.982726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980715 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17ee44c6-7ddd-473c-8b5b-c2e908141e09-ovnkube-config\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.982726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.980795 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a44888c-a9bf-4378-8785-26bc77ef43bb-tmp-dir\") pod \"node-resolver-4vxw9\" (UID: \"4a44888c-a9bf-4378-8785-26bc77ef43bb\") " pod="openshift-dns/node-resolver-4vxw9" Sep 18 17:14:50.983129 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.982773 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-tmp\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.983129 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.982917 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-etc-tuned\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.983129 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.982935 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17ee44c6-7ddd-473c-8b5b-c2e908141e09-ovn-node-metrics-cert\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:50.990322 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.990239 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdhrk\" (UniqueName: \"kubernetes.io/projected/39b8b8ec-f31c-4f91-94a9-b552c0f2c691-kube-api-access-bdhrk\") pod \"tuned-jtcp7\" (UID: \"39b8b8ec-f31c-4f91-94a9-b552c0f2c691\") " pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:50.993451 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.993426 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmcc9\" (UniqueName: \"kubernetes.io/projected/645e614d-3ae7-4057-9b97-e7386685af67-kube-api-access-gmcc9\") pod \"iptables-alerter-5fdzk\" (UID: \"645e614d-3ae7-4057-9b97-e7386685af67\") " pod="openshift-network-operator/iptables-alerter-5fdzk" Sep 18 17:14:50.993859 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.993838 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48mnv\" (UniqueName: \"kubernetes.io/projected/4a44888c-a9bf-4378-8785-26bc77ef43bb-kube-api-access-48mnv\") pod \"node-resolver-4vxw9\" (UID: \"4a44888c-a9bf-4378-8785-26bc77ef43bb\") " pod="openshift-dns/node-resolver-4vxw9" Sep 18 17:14:50.993979 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:50.993953 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vgxz\" (UniqueName: \"kubernetes.io/projected/17ee44c6-7ddd-473c-8b5b-c2e908141e09-kube-api-access-4vgxz\") pod \"ovnkube-node-qf4rc\" (UID: \"17ee44c6-7ddd-473c-8b5b-c2e908141e09\") " pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:51.063347 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.063310 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" Sep 18 17:14:51.071123 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.071094 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-44qw2" Sep 18 17:14:51.080738 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.080716 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x7ns6" Sep 18 17:14:51.086097 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.086080 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s2ln6" Sep 18 17:14:51.092480 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.092459 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-78r8w" Sep 18 17:14:51.098979 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.098955 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" Sep 18 17:14:51.104732 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.104714 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4vxw9" Sep 18 17:14:51.111357 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.111337 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5fdzk" Sep 18 17:14:51.116184 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.116168 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:14:51.382150 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.382062 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:51.382327 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:51.382250 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:14:51.382387 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:51.382369 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs podName:cd4d7977-637e-4068-8d44-515534534834 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:52.382348358 +0000 UTC m=+5.635509158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs") pod "network-metrics-daemon-clj2d" (UID: "cd4d7977-637e-4068-8d44-515534534834") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:14:51.483512 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.483466 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:51.483691 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.483533 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzxm\" (UniqueName: \"kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm\") pod \"network-check-target-wqx8x\" (UID: \"6883d049-759d-4fc5-af31-d30305f4b092\") " pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:14:51.483691 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:51.483634 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 17:14:51.483810 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:51.483713 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret podName:af831002-a9cc-4239-9821-98618e18e764 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:52.483698218 +0000 UTC m=+5.736859012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret") pod "global-pull-secret-syncer-qrbrr" (UID: "af831002-a9cc-4239-9821-98618e18e764") : object "kube-system"/"original-pull-secret" not registered Sep 18 17:14:51.483810 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:51.483715 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:14:51.483810 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:51.483738 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:14:51.483810 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:51.483747 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dkzxm for pod openshift-network-diagnostics/network-check-target-wqx8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:14:51.483810 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:51.483793 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm podName:6883d049-759d-4fc5-af31-d30305f4b092 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:52.483780538 +0000 UTC m=+5.736941337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkzxm" (UniqueName: "kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm") pod "network-check-target-wqx8x" (UID: "6883d049-759d-4fc5-af31-d30305f4b092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:14:51.706240 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.706154 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 18 17:14:51.756219 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:51.756180 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51a62b4_6c7c_4e29_ae8c_4f22d4d66f73.slice/crio-b805c64961f3b34db88cfd30c38f333393a705e6a64c2deb9256faebab410945 WatchSource:0}: Error finding container b805c64961f3b34db88cfd30c38f333393a705e6a64c2deb9256faebab410945: Status 404 returned error can't find the container with id b805c64961f3b34db88cfd30c38f333393a705e6a64c2deb9256faebab410945 Sep 18 17:14:51.757989 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:51.757825 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb69917_471e_4d0a_bba9_60d8552d49fb.slice/crio-8454d03fd2fef89c1fc8b94f286aecacc97d70fef3f6ea7353f05add0ca62b52 WatchSource:0}: Error finding container 8454d03fd2fef89c1fc8b94f286aecacc97d70fef3f6ea7353f05add0ca62b52: Status 404 returned error can't find the container with id 8454d03fd2fef89c1fc8b94f286aecacc97d70fef3f6ea7353f05add0ca62b52 Sep 18 17:14:51.761828 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:51.761798 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ee44c6_7ddd_473c_8b5b_c2e908141e09.slice/crio-610bf170db788a6940c55631b504594f862dfa87b1486428408d712938cc507f WatchSource:0}: Error finding container 610bf170db788a6940c55631b504594f862dfa87b1486428408d712938cc507f: Status 404 returned error can't find the container with id 610bf170db788a6940c55631b504594f862dfa87b1486428408d712938cc507f Sep 18 17:14:51.762537 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:14:51.762499 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156fc6c4_1430_428e_aa27_fb144f9ab893.slice/crio-26f8afeeca6ddba1edaa992f85e142c0541362f5357c10f4d1ea313f38514561 WatchSource:0}: Error finding container 26f8afeeca6ddba1edaa992f85e142c0541362f5357c10f4d1ea313f38514561: Status 404 returned error can't find the container with id 26f8afeeca6ddba1edaa992f85e142c0541362f5357c10f4d1ea313f38514561 Sep 18 17:14:51.826107 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.826056 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 17:09:50 +0000 UTC" deadline="2028-05-03 02:26:15.59576207 +0000 UTC" Sep 18 17:14:51.826107 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.826106 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14217h11m23.769660016s" Sep 18 17:14:51.858844 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.858816 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:14:51.858964 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:51.858940 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:14:51.859038 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.858981 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:51.859091 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:51.859058 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:14:51.870426 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.870393 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-9.ec2.internal" event={"ID":"e55b801a498c78c600fd5d152605bab9","Type":"ContainerStarted","Data":"fb26409af85bca39952eae067e291be00e6c42a877ca9870df4f2f47e35ad672"} Sep 18 17:14:51.871631 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.871600 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s2ln6" event={"ID":"03103480-84b5-46bf-a5ac-03f8aac2216a","Type":"ContainerStarted","Data":"f27f5473c69b8b703d3e2a54b0e5431ab0702d5bc4b9915354fc3226a6ecb77e"} Sep 18 17:14:51.872774 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.872747 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-78r8w" event={"ID":"156fc6c4-1430-428e-aa27-fb144f9ab893","Type":"ContainerStarted","Data":"26f8afeeca6ddba1edaa992f85e142c0541362f5357c10f4d1ea313f38514561"} Sep 18 17:14:51.873978 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.873958 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" event={"ID":"6cb69917-471e-4d0a-bba9-60d8552d49fb","Type":"ContainerStarted","Data":"8454d03fd2fef89c1fc8b94f286aecacc97d70fef3f6ea7353f05add0ca62b52"} Sep 18 17:14:51.875127 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.875103 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-44qw2" event={"ID":"b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73","Type":"ContainerStarted","Data":"b805c64961f3b34db88cfd30c38f333393a705e6a64c2deb9256faebab410945"} Sep 18 17:14:51.876224 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.876204 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5fdzk" event={"ID":"645e614d-3ae7-4057-9b97-e7386685af67","Type":"ContainerStarted","Data":"e1bf9a10dd2fc1f39f33ba36b7a7c311aee948a1267827ea1840408ca0ce2e9f"} Sep 18 17:14:51.877349 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.877325 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" event={"ID":"39b8b8ec-f31c-4f91-94a9-b552c0f2c691","Type":"ContainerStarted","Data":"fdbaa5d4039a58736ad227312ee1fbd6493e777d3af6789cc9da9c26a75224b3"} Sep 18 17:14:51.878457 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.878436 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4vxw9" event={"ID":"4a44888c-a9bf-4378-8785-26bc77ef43bb","Type":"ContainerStarted","Data":"bd5ac3d54a9d7cf82d5bf16aa4cfcc0ce748d4f80a9d92840ea466f0e83702d6"} Sep 18 17:14:51.879508 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.879424 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7ns6" event={"ID":"492a5d0b-c698-4eab-b0d6-5c2961725674","Type":"ContainerStarted","Data":"e0787bac38190d3bb7cedec7b418eb515ff47088b67c321a07806a32a1bc514e"} Sep 18 17:14:51.880637 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.880617 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" event={"ID":"17ee44c6-7ddd-473c-8b5b-c2e908141e09","Type":"ContainerStarted","Data":"610bf170db788a6940c55631b504594f862dfa87b1486428408d712938cc507f"} Sep 18 17:14:51.888749 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:51.888707 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-9.ec2.internal" podStartSLOduration=1.888689163 podStartE2EDuration="1.888689163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:14:51.888611548 +0000 UTC m=+5.141772366" watchObservedRunningTime="2026-09-18 17:14:51.888689163 +0000 UTC m=+5.141849979" Sep 18 17:14:52.391044 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:52.390851 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:52.391209 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:52.391052 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:14:52.391209 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:52.391116 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs podName:cd4d7977-637e-4068-8d44-515534534834 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:54.391096591 +0000 UTC m=+7.644257393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs") pod "network-metrics-daemon-clj2d" (UID: "cd4d7977-637e-4068-8d44-515534534834") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:14:52.492267 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:52.492229 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:52.492443 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:52.492311 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzxm\" (UniqueName: \"kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm\") pod \"network-check-target-wqx8x\" (UID: \"6883d049-759d-4fc5-af31-d30305f4b092\") " pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:14:52.492515 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:52.492506 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:14:52.492570 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:52.492523 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:14:52.492570 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:52.492537 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dkzxm for pod openshift-network-diagnostics/network-check-target-wqx8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:14:52.492674 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:52.492600 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm podName:6883d049-759d-4fc5-af31-d30305f4b092 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:54.492580348 +0000 UTC m=+7.745741146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkzxm" (UniqueName: "kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm") pod "network-check-target-wqx8x" (UID: "6883d049-759d-4fc5-af31-d30305f4b092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:14:52.493110 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:52.493091 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 17:14:52.493189 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:52.493156 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret podName:af831002-a9cc-4239-9821-98618e18e764 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:54.493141255 +0000 UTC m=+7.746302056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret") pod "global-pull-secret-syncer-qrbrr" (UID: "af831002-a9cc-4239-9821-98618e18e764") : object "kube-system"/"original-pull-secret" not registered Sep 18 17:14:52.826857 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:52.826766 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 17:09:50 +0000 UTC" deadline="2028-03-01 08:28:50.986669488 +0000 UTC" Sep 18 17:14:52.826857 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:52.826807 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12711h13m58.159867093s" Sep 18 17:14:52.858871 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:52.858836 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:52.859019 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:52.858977 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:14:52.887907 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:52.887664 2560 generic.go:356] "Generic (PLEG): container finished" podID="eccee8496a5198ea9303df36f6adb7ef" containerID="0b2595cc9653bd10f58c8cb3e705cbc2417acd99f98235016c60ae458768f571" exitCode=0 Sep 18 17:14:52.888527 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:52.888202 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal" event={"ID":"eccee8496a5198ea9303df36f6adb7ef","Type":"ContainerDied","Data":"0b2595cc9653bd10f58c8cb3e705cbc2417acd99f98235016c60ae458768f571"} Sep 18 17:14:53.858669 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:53.858638 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:14:53.859106 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:53.858760 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:14:53.859213 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:53.859194 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:53.859425 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:53.859321 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:14:53.907776 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:53.907737 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal" event={"ID":"eccee8496a5198ea9303df36f6adb7ef","Type":"ContainerStarted","Data":"a8d4419816eeb02f365c8423fa5d7b5d4b2e5fec81a740a91a934ef714f432cd"} Sep 18 17:14:53.924440 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:53.924394 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-9.ec2.internal" podStartSLOduration=3.924377926 podStartE2EDuration="3.924377926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:14:53.923112341 +0000 UTC m=+7.176273180" watchObservedRunningTime="2026-09-18 17:14:53.924377926 +0000 UTC m=+7.177538744" Sep 18 17:14:54.409038 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:54.409001 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:54.409225 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:54.409191 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:14:54.409305 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:54.409254 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs podName:cd4d7977-637e-4068-8d44-515534534834 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:58.409234518 +0000 UTC m=+11.662395318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs") pod "network-metrics-daemon-clj2d" (UID: "cd4d7977-637e-4068-8d44-515534534834") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:14:54.510420 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:54.510383 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:54.510574 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:54.510438 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzxm\" (UniqueName: \"kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm\") pod \"network-check-target-wqx8x\" (UID: \"6883d049-759d-4fc5-af31-d30305f4b092\") " pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:14:54.510642 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:54.510593 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:14:54.510642 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:54.510612 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:14:54.510642 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:54.510626 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dkzxm for pod openshift-network-diagnostics/network-check-target-wqx8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:14:54.510789 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:54.510682 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm podName:6883d049-759d-4fc5-af31-d30305f4b092 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:58.510664823 +0000 UTC m=+11.763825624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkzxm" (UniqueName: "kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm") pod "network-check-target-wqx8x" (UID: "6883d049-759d-4fc5-af31-d30305f4b092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:14:54.511154 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:54.511134 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 17:14:54.511225 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:54.511186 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret podName:af831002-a9cc-4239-9821-98618e18e764 nodeName:}" failed. No retries permitted until 2026-09-18 17:14:58.511171911 +0000 UTC m=+11.764332720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret") pod "global-pull-secret-syncer-qrbrr" (UID: "af831002-a9cc-4239-9821-98618e18e764") : object "kube-system"/"original-pull-secret" not registered Sep 18 17:14:54.858822 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:54.858743 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:54.859236 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:54.858882 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:14:55.858684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:55.858646 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:14:55.858860 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:55.858788 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:14:55.859262 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:55.859179 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:55.859345 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:55.859284 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:14:56.859213 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:56.859158 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:56.859697 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:56.859313 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:14:57.859241 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:57.858577 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:14:57.859241 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:57.858716 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:14:57.859241 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:57.859141 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:57.859241 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:57.859241 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:14:58.446061 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:58.445924 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:58.446250 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:58.446087 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:14:58.446250 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:58.446151 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs podName:cd4d7977-637e-4068-8d44-515534534834 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:06.446132625 +0000 UTC m=+19.699293437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs") pod "network-metrics-daemon-clj2d" (UID: "cd4d7977-637e-4068-8d44-515534534834") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:14:58.546935 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:58.546886 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:58.546935 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:58.546938 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzxm\" (UniqueName: \"kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm\") pod \"network-check-target-wqx8x\" (UID: \"6883d049-759d-4fc5-af31-d30305f4b092\") " pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:14:58.547212 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:58.547075 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:14:58.547212 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:58.547091 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:14:58.547212 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:58.547101 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dkzxm for pod openshift-network-diagnostics/network-check-target-wqx8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:14:58.547212 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:58.547152 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm podName:6883d049-759d-4fc5-af31-d30305f4b092 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:06.547138544 +0000 UTC m=+19.800299340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkzxm" (UniqueName: "kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm") pod "network-check-target-wqx8x" (UID: "6883d049-759d-4fc5-af31-d30305f4b092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:14:58.547212 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:58.547159 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 17:14:58.547505 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:58.547225 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret podName:af831002-a9cc-4239-9821-98618e18e764 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:06.547208046 +0000 UTC m=+19.800368860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret") pod "global-pull-secret-syncer-qrbrr" (UID: "af831002-a9cc-4239-9821-98618e18e764") : object "kube-system"/"original-pull-secret" not registered Sep 18 17:14:58.859534 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:58.859454 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:14:58.859930 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:58.859568 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:14:59.858423 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:59.858341 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:14:59.858594 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:14:59.858346 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:14:59.858594 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:59.858470 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:14:59.858594 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:14:59.858574 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:15:00.858829 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:00.858799 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:00.859264 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:00.858923 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:15:01.858582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:01.858540 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:01.858759 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:01.858659 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:15:01.858759 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:01.858719 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:01.858863 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:01.858841 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:15:02.858694 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:02.858666 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:02.858848 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:02.858795 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:15:03.858677 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:03.858636 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:03.859124 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:03.858654 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:03.859124 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:03.858772 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:15:03.859124 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:03.858871 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:15:04.858380 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:04.858349 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:04.858566 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:04.858474 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:15:05.858216 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:05.858187 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:05.858608 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:05.858227 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:05.858608 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:05.858340 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:15:05.858608 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:05.858411 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:15:06.504503 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:06.504467 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:06.504681 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:06.504601 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:15:06.504681 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:06.504665 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs podName:cd4d7977-637e-4068-8d44-515534534834 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:22.504644199 +0000 UTC m=+35.757804997 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs") pod "network-metrics-daemon-clj2d" (UID: "cd4d7977-637e-4068-8d44-515534534834") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 18 17:15:06.605630 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:06.605582 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:06.605630 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:06.605633 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzxm\" (UniqueName: \"kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm\") pod \"network-check-target-wqx8x\" (UID: \"6883d049-759d-4fc5-af31-d30305f4b092\") " pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:06.605869 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:06.605727 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 18 17:15:06.605869 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:06.605748 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:15:06.605869 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:06.605766 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:15:06.605869 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:06.605778 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dkzxm for pod openshift-network-diagnostics/network-check-target-wqx8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:15:06.605869 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:06.605801 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret podName:af831002-a9cc-4239-9821-98618e18e764 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:22.605786403 +0000 UTC m=+35.858947212 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret") pod "global-pull-secret-syncer-qrbrr" (UID: "af831002-a9cc-4239-9821-98618e18e764") : object "kube-system"/"original-pull-secret" not registered Sep 18 17:15:06.605869 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:06.605827 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm podName:6883d049-759d-4fc5-af31-d30305f4b092 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:22.605811401 +0000 UTC m=+35.858972211 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkzxm" (UniqueName: "kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm") pod "network-check-target-wqx8x" (UID: "6883d049-759d-4fc5-af31-d30305f4b092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:15:06.858774 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:06.858699 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:06.859206 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:06.858822 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:15:07.858841 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:07.858802 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:07.859245 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:07.858855 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:07.859245 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:07.858942 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:15:07.859245 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:07.859081 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:15:08.859388 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:08.859367 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:08.859737 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:08.859473 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:15:09.858397 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.857820 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:09.858397 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.857858 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:09.858397 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:09.858363 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:15:09.858647 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:09.858479 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:15:09.951124 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.951099 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s2ln6" event={"ID":"03103480-84b5-46bf-a5ac-03f8aac2216a","Type":"ContainerStarted","Data":"5299df792af440ac2d975dc7e67b01c968ff4f6abca0a1e733d49a4b665325b1"} Sep 18 17:15:09.952647 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.952620 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-78r8w" event={"ID":"156fc6c4-1430-428e-aa27-fb144f9ab893","Type":"ContainerStarted","Data":"5be1e705073410d4f9490994360ccc7bee930930680c87df182dac2b26e8b0db"} Sep 18 17:15:09.954062 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.954040 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" event={"ID":"6cb69917-471e-4d0a-bba9-60d8552d49fb","Type":"ContainerStarted","Data":"5b3f22d620c1ee10528ad4c35015e01f14c6aeae21fde1f65247cfb546f51c8e"} Sep 18 17:15:09.955424 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.955403 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-44qw2" event={"ID":"b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73","Type":"ContainerStarted","Data":"f361e47d96f1ec9501cf7e8f83fbca860b66577d93ec1174170c08956b255457"} Sep 18 17:15:09.956737 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.956707 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" event={"ID":"39b8b8ec-f31c-4f91-94a9-b552c0f2c691","Type":"ContainerStarted","Data":"4e11c8ed3dc58c1a0a0d8ee498e03fda1609a5c973d172f25aa2379f39f4b36d"} Sep 18 17:15:09.958372 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.958348 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4vxw9" event={"ID":"4a44888c-a9bf-4378-8785-26bc77ef43bb","Type":"ContainerStarted","Data":"009221844becdec21e7b06f74dfe6844dff0f8586ef9ef0585785f7b42c6b6b4"} Sep 18 17:15:09.960838 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.960811 2560 generic.go:356] "Generic (PLEG): container finished" podID="492a5d0b-c698-4eab-b0d6-5c2961725674" containerID="14fe17060eb9e853ef703f1c365bd057ae3961070c755306a34dc647ebb675d3" exitCode=0 Sep 18 17:15:09.960936 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.960895 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7ns6" event={"ID":"492a5d0b-c698-4eab-b0d6-5c2961725674","Type":"ContainerDied","Data":"14fe17060eb9e853ef703f1c365bd057ae3961070c755306a34dc647ebb675d3"} Sep 18 17:15:09.965949 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.965927 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:15:09.966364 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.966339 2560 generic.go:356] "Generic (PLEG): container finished" podID="17ee44c6-7ddd-473c-8b5b-c2e908141e09" containerID="8831dc4ac7cc7072d8ecbeff241ce5971f64c22c4a4d340dac88bf2b36110956" exitCode=1 Sep 18 17:15:09.966446 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.966374 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" event={"ID":"17ee44c6-7ddd-473c-8b5b-c2e908141e09","Type":"ContainerStarted","Data":"04260d3437887672c6c423691dba7d2e9361617b35413ce2490584c40290367c"} Sep 18 17:15:09.966516 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.966456 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" event={"ID":"17ee44c6-7ddd-473c-8b5b-c2e908141e09","Type":"ContainerStarted","Data":"e076e8fb78d8882fadc162306fd96f40a49a487e9ca27e73dbaedf073dbff4b7"} Sep 18 17:15:09.966516 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.966508 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" event={"ID":"17ee44c6-7ddd-473c-8b5b-c2e908141e09","Type":"ContainerStarted","Data":"5bb6083a5639225018bff725d0a3409148ddb56ce8886dd8c9cdb036230c8b9c"} Sep 18 17:15:09.966598 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.966557 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" event={"ID":"17ee44c6-7ddd-473c-8b5b-c2e908141e09","Type":"ContainerStarted","Data":"dc8bded6f3d87a9e2cafc6e8f16aaece206a40ad884dbc46e40110ec6883f7f0"} Sep 18 17:15:09.966649 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.966605 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" event={"ID":"17ee44c6-7ddd-473c-8b5b-c2e908141e09","Type":"ContainerDied","Data":"8831dc4ac7cc7072d8ecbeff241ce5971f64c22c4a4d340dac88bf2b36110956"} Sep 18 17:15:09.966687 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.966655 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" event={"ID":"17ee44c6-7ddd-473c-8b5b-c2e908141e09","Type":"ContainerStarted","Data":"cb2a91144b1af9126ac98190a8ec1d32841a8e43e32c993c2da0eccf5b0a20d2"} Sep 18 17:15:09.968804 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.968767 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s2ln6" podStartSLOduration=4.881192714 podStartE2EDuration="21.968754842s" totalImagesPullingTime="17.087562128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:09.968137441 +0000 UTC m=+23.221298277" watchObservedRunningTime="2026-09-18 17:15:09.968754842 +0000 UTC m=+23.221915940" Sep 18 17:15:09.982670 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:09.982622 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-44qw2" podStartSLOduration=4.90646808 podStartE2EDuration="21.982606773s" totalImagesPullingTime="17.076138693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:09.982585081 +0000 UTC m=+23.235745897" watchObservedRunningTime="2026-09-18 17:15:09.982606773 +0000 UTC m=+23.235767592" Sep 18 17:15:10.021801 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.021757 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-78r8w" podStartSLOduration=4.044677919 podStartE2EDuration="21.02174237s" totalImagesPullingTime="16.977064451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:10.021651769 +0000 UTC m=+23.274812597" watchObservedRunningTime="2026-09-18 17:15:10.02174237 +0000 UTC m=+23.274903186" Sep 18 17:15:10.038742 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.038694 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4vxw9" podStartSLOduration=3.9757314299999997 podStartE2EDuration="21.038680649s" totalImagesPullingTime="17.062949219s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:10.037883392 +0000 UTC m=+23.291044210" watchObservedRunningTime="2026-09-18 17:15:10.038680649 +0000 UTC m=+23.291841467" Sep 18 17:15:10.095895 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.095848 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jtcp7" podStartSLOduration=4.053928176 podStartE2EDuration="21.095833272s" totalImagesPullingTime="17.041905096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:10.094631692 +0000 UTC m=+23.347792521" watchObservedRunningTime="2026-09-18 17:15:10.095833272 +0000 UTC m=+23.348994088" Sep 18 17:15:10.154785 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.154759 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 18 17:15:10.204073 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.204044 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:10.267834 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.267802 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:10.431790 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.431712 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:10.532917 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.532885 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:10.839148 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.838994 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T17:15:10.154777861Z","UUID":"e0e848b0-423d-4c91-acd8-0c37e2ab1875","Handler":null,"Name":"","Endpoint":""} Sep 18 17:15:10.841725 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.841704 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 18 17:15:10.841855 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.841742 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 18 17:15:10.858812 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.858771 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:10.858945 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:10.858908 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:15:10.971860 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.971822 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" event={"ID":"6cb69917-471e-4d0a-bba9-60d8552d49fb","Type":"ContainerStarted","Data":"0aaee4d8690b4f4a57b9489ea883779d430b2fd36b8e4e5f6816908f043e9ea2"} Sep 18 17:15:10.975600 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.974442 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5fdzk" event={"ID":"645e614d-3ae7-4057-9b97-e7386685af67","Type":"ContainerStarted","Data":"2d0cd4e6192fc35899dedf1c434e9472630dc9b6adb8ecd49ff82dc4f19ec0c8"} Sep 18 17:15:10.992529 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:10.992488 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5fdzk" podStartSLOduration=5.0154003 podStartE2EDuration="21.992474311s" totalImagesPullingTime="16.977074011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:10.992121228 +0000 UTC m=+24.245282046" watchObservedRunningTime="2026-09-18 17:15:10.992474311 +0000 UTC m=+24.245635139" Sep 18 17:15:11.037390 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:11.037364 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:11.858062 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:11.858034 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:11.858191 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:11.858141 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:15:11.858191 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:11.858175 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:11.858321 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:11.858244 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:15:11.979313 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:11.979260 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" event={"ID":"6cb69917-471e-4d0a-bba9-60d8552d49fb","Type":"ContainerStarted","Data":"d6267247f8fc50ccc00a272fd5f02ec0e59d36e6a2ee333f745b31b5abe03085"} Sep 18 17:15:11.983574 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:11.983555 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:15:11.983996 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:11.983923 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" event={"ID":"17ee44c6-7ddd-473c-8b5b-c2e908141e09","Type":"ContainerStarted","Data":"ce3da4925676e38b7c13a667d05d3d7f2b3155d74724a8db132e1a82c1c134d8"} Sep 18 17:15:11.999591 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:11.999546 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-klqz5" podStartSLOduration=4.929058375 podStartE2EDuration="23.999529265s" totalImagesPullingTime="19.07047089s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:11.999282062 +0000 UTC m=+25.252442880" watchObservedRunningTime="2026-09-18 17:15:11.999529265 +0000 UTC m=+25.252690083" Sep 18 17:15:12.046523 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:12.046455 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:12.108427 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:12.108397 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:12.239796 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:12.239767 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:12.635884 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:12.635849 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:12.737223 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:12.737190 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:12.858479 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:12.858451 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:12.858633 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:12.858550 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:15:13.858642 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:13.858456 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:13.859111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:13.858456 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:13.859111 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:13.858766 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:15:13.859111 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:13.858812 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:15:14.853584 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:14.853414 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-78r8w" Sep 18 17:15:14.854407 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:14.854190 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-78r8w" Sep 18 17:15:14.858491 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:14.858370 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:14.858491 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:14.858471 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:15:14.992879 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:14.992854 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:15:14.993581 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:14.993183 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" event={"ID":"17ee44c6-7ddd-473c-8b5b-c2e908141e09","Type":"ContainerStarted","Data":"191264b538e8f68535832aebd5f8e59d8da117c74169d972ea73d837139442ef"} Sep 18 17:15:14.993581 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:14.993492 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-78r8w" Sep 18 17:15:14.993703 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:14.993684 2560 scope.go:117] "RemoveContainer" containerID="8831dc4ac7cc7072d8ecbeff241ce5971f64c22c4a4d340dac88bf2b36110956" Sep 18 17:15:14.994089 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:14.994070 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-78r8w" Sep 18 17:15:15.092098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:15.092073 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:15.858576 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:15.858548 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:15.858726 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:15.858648 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:15:15.858726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:15.858702 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:15.858840 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:15.858791 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:15:15.996981 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:15.996943 2560 generic.go:356] "Generic (PLEG): container finished" podID="492a5d0b-c698-4eab-b0d6-5c2961725674" containerID="8522a49b4aa9e54a3d42f9659bc93d1e99273c5dc3969e1bdeab3c2efcc25306" exitCode=0 Sep 18 17:15:15.997577 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:15.997033 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7ns6" event={"ID":"492a5d0b-c698-4eab-b0d6-5c2961725674","Type":"ContainerDied","Data":"8522a49b4aa9e54a3d42f9659bc93d1e99273c5dc3969e1bdeab3c2efcc25306"} Sep 18 17:15:16.000876 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.000859 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:15:16.001314 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.001284 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" event={"ID":"17ee44c6-7ddd-473c-8b5b-c2e908141e09","Type":"ContainerStarted","Data":"4c656dc1c5851f612c42eda35b1df6cf6d18132a445afaf76164270ea4e3816a"} Sep 18 17:15:16.001723 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.001690 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:15:16.001823 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.001778 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:15:16.001881 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.001829 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:15:16.016134 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.016078 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:15:16.016232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.016201 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:15:16.060160 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.060106 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" podStartSLOduration=9.747634379 podStartE2EDuration="27.060086719s" totalImagesPullingTime="17.31245234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:16.05951845 +0000 UTC m=+29.312679300" watchObservedRunningTime="2026-09-18 17:15:16.060086719 +0000 UTC m=+29.313247538" Sep 18 17:15:16.743137 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.742918 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qrbrr"] Sep 18 17:15:16.743334 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.743255 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:16.743420 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:16.743393 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:15:16.746940 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.746909 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wqx8x"] Sep 18 17:15:16.747061 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.746991 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:16.747123 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:16.747079 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:15:16.747740 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.747679 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-clj2d"] Sep 18 17:15:16.747855 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:16.747765 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:16.747911 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:16.747855 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:15:17.858688 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:17.858654 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:17.859207 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:17.858654 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:17.859207 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:17.858782 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:15:17.859207 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:17.858832 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:15:18.008179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:18.008154 2560 generic.go:356] "Generic (PLEG): container finished" podID="492a5d0b-c698-4eab-b0d6-5c2961725674" containerID="bc669d5f11227b516f8d9e167ea354091ace6935d70a35547cc1da3bd950a45a" exitCode=0 Sep 18 17:15:18.008309 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:18.008236 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7ns6" event={"ID":"492a5d0b-c698-4eab-b0d6-5c2961725674","Type":"ContainerDied","Data":"bc669d5f11227b516f8d9e167ea354091ace6935d70a35547cc1da3bd950a45a"} Sep 18 17:15:18.858548 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:18.858528 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:18.858664 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:18.858627 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:15:19.013453 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:19.013424 2560 generic.go:356] "Generic (PLEG): container finished" podID="492a5d0b-c698-4eab-b0d6-5c2961725674" containerID="4f6dbb23ae37faa4f8ef790e3565f4cb01970fe51d3243c6b30945c69afda9d9" exitCode=0 Sep 18 17:15:19.013845 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:19.013488 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7ns6" event={"ID":"492a5d0b-c698-4eab-b0d6-5c2961725674","Type":"ContainerDied","Data":"4f6dbb23ae37faa4f8ef790e3565f4cb01970fe51d3243c6b30945c69afda9d9"} Sep 18 17:15:19.858106 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:19.858071 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:19.858281 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:19.858082 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:19.858281 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:19.858202 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qrbrr" podUID="af831002-a9cc-4239-9821-98618e18e764" Sep 18 17:15:19.858407 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:19.858279 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-clj2d" podUID="cd4d7977-637e-4068-8d44-515534534834" Sep 18 17:15:20.857859 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:20.857826 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:20.858202 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:20.857949 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqx8x" podUID="6883d049-759d-4fc5-af31-d30305f4b092" Sep 18 17:15:21.587661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.587628 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-9.ec2.internal" event="NodeReady" Sep 18 17:15:21.587832 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.587766 2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 18 17:15:21.641773 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.641746 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qqglr"] Sep 18 17:15:21.669457 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.668613 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k8tvl"] Sep 18 17:15:21.669457 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.668753 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:21.672375 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.672150 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 18 17:15:21.672375 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.672208 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 18 17:15:21.672566 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.672482 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99knh\"" Sep 18 17:15:21.684779 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.684713 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qqglr"] Sep 18 17:15:21.684877 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.684785 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k8tvl" Sep 18 17:15:21.684877 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.684806 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k8tvl"] Sep 18 17:15:21.688133 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.688100 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 18 17:15:21.688403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.688243 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ddjzh\"" Sep 18 17:15:21.688403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.688107 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 18 17:15:21.688403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.688343 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 18 17:15:21.807493 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.807453 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:21.817208 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.817180 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e62a1939-824b-49c5-857a-c0c41fb3c31a-cert\") pod \"ingress-canary-k8tvl\" (UID: \"e62a1939-824b-49c5-857a-c0c41fb3c31a\") " pod="openshift-ingress-canary/ingress-canary-k8tvl" Sep 18 17:15:21.817367 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.817219 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-config-volume\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:21.817367 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.817263 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:21.817539 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.817301 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8q5s\" (UniqueName: \"kubernetes.io/projected/e62a1939-824b-49c5-857a-c0c41fb3c31a-kube-api-access-c8q5s\") pod \"ingress-canary-k8tvl\" (UID: \"e62a1939-824b-49c5-857a-c0c41fb3c31a\") " pod="openshift-ingress-canary/ingress-canary-k8tvl" Sep 18 17:15:21.817760 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.817724 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-tmp-dir\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:21.817852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.817777 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt5zz\" (UniqueName: \"kubernetes.io/projected/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-kube-api-access-qt5zz\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:21.858687 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.858654 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:21.859105 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.858701 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:21.862553 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.862532 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 18 17:15:21.862908 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.862837 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 18 17:15:21.863021 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.862986 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2h6n2\"" Sep 18 17:15:21.900034 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.900014 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:21.918085 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.918067 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e62a1939-824b-49c5-857a-c0c41fb3c31a-cert\") pod \"ingress-canary-k8tvl\" (UID: \"e62a1939-824b-49c5-857a-c0c41fb3c31a\") " pod="openshift-ingress-canary/ingress-canary-k8tvl" Sep 18 17:15:21.918154 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.918108 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-config-volume\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:21.918154 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.918142 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:21.918226 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.918161 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8q5s\" (UniqueName: \"kubernetes.io/projected/e62a1939-824b-49c5-857a-c0c41fb3c31a-kube-api-access-c8q5s\") pod \"ingress-canary-k8tvl\" (UID: \"e62a1939-824b-49c5-857a-c0c41fb3c31a\") " pod="openshift-ingress-canary/ingress-canary-k8tvl" Sep 18 17:15:21.918226 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.918183 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-tmp-dir\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:21.918226 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.918199 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt5zz\" (UniqueName: \"kubernetes.io/projected/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-kube-api-access-qt5zz\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:21.918374 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:21.918287 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 17:15:21.918374 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:21.918362 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls podName:ddf535f2-88b7-40b0-8d49-c9bfe97b3679 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:22.418342378 +0000 UTC m=+35.671503185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls") pod "dns-default-qqglr" (UID: "ddf535f2-88b7-40b0-8d49-c9bfe97b3679") : secret "dns-default-metrics-tls" not found Sep 18 17:15:21.918623 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.918603 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-tmp-dir\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:21.918770 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.918746 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-config-volume\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:21.922019 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.921997 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e62a1939-824b-49c5-857a-c0c41fb3c31a-cert\") pod \"ingress-canary-k8tvl\" (UID: \"e62a1939-824b-49c5-857a-c0c41fb3c31a\") " pod="openshift-ingress-canary/ingress-canary-k8tvl" Sep 18 17:15:21.929398 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.929372 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8q5s\" (UniqueName: \"kubernetes.io/projected/e62a1939-824b-49c5-857a-c0c41fb3c31a-kube-api-access-c8q5s\") pod \"ingress-canary-k8tvl\" (UID: \"e62a1939-824b-49c5-857a-c0c41fb3c31a\") " pod="openshift-ingress-canary/ingress-canary-k8tvl" Sep 18 17:15:21.930042 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.930027 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt5zz\" (UniqueName: \"kubernetes.io/projected/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-kube-api-access-qt5zz\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:21.994920 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:21.994887 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k8tvl" Sep 18 17:15:22.310542 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:22.310463 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k8tvl"] Sep 18 17:15:22.422238 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:22.422205 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:22.422398 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:22.422380 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 17:15:22.422467 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:22.422454 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls podName:ddf535f2-88b7-40b0-8d49-c9bfe97b3679 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:23.422432571 +0000 UTC m=+36.675593366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls") pod "dns-default-qqglr" (UID: "ddf535f2-88b7-40b0-8d49-c9bfe97b3679") : secret "dns-default-metrics-tls" not found Sep 18 17:15:22.523033 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:22.522995 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:22.523215 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:22.523134 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 18 17:15:22.523215 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:22.523215 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs podName:cd4d7977-637e-4068-8d44-515534534834 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:54.523182061 +0000 UTC m=+67.776342857 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs") pod "network-metrics-daemon-clj2d" (UID: "cd4d7977-637e-4068-8d44-515534534834") : secret "metrics-daemon-secret" not found Sep 18 17:15:22.623424 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:22.623339 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:22.623424 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:22.623382 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzxm\" (UniqueName: \"kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm\") pod \"network-check-target-wqx8x\" (UID: \"6883d049-759d-4fc5-af31-d30305f4b092\") " pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:22.623622 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:22.623509 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 18 17:15:22.623622 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:22.623529 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 18 17:15:22.623622 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:22.623542 2560 projected.go:196] Error preparing data for projected volume kube-api-access-dkzxm for pod openshift-network-diagnostics/network-check-target-wqx8x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:15:22.623622 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:22.623614 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm podName:6883d049-759d-4fc5-af31-d30305f4b092 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:54.623597776 +0000 UTC m=+67.876758588 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dkzxm" (UniqueName: "kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm") pod "network-check-target-wqx8x" (UID: "6883d049-759d-4fc5-af31-d30305f4b092") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 18 17:15:22.626205 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:22.626175 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/af831002-a9cc-4239-9821-98618e18e764-original-pull-secret\") pod \"global-pull-secret-syncer-qrbrr\" (UID: \"af831002-a9cc-4239-9821-98618e18e764\") " pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:22.771010 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:22.770964 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qrbrr" Sep 18 17:15:22.858738 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:22.858704 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:22.861804 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:22.861775 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 18 17:15:22.861931 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:22.861830 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 18 17:15:22.861931 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:22.861893 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f2lgv\"" Sep 18 17:15:23.237403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:23.237369 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:23.301937 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:23.301900 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:23.430006 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:23.429973 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:23.430219 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:23.430145 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 17:15:23.430219 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:23.430217 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls podName:ddf535f2-88b7-40b0-8d49-c9bfe97b3679 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:25.43019413 +0000 UTC m=+38.683354927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls") pod "dns-default-qqglr" (UID: "ddf535f2-88b7-40b0-8d49-c9bfe97b3679") : secret "dns-default-metrics-tls" not found Sep 18 17:15:23.743015 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:15:23.742988 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62a1939_824b_49c5_857a_c0c41fb3c31a.slice/crio-db0739ad564d2223b458634f18979607df145900f04a94a03858b770287d85ba WatchSource:0}: Error finding container db0739ad564d2223b458634f18979607df145900f04a94a03858b770287d85ba: Status 404 returned error can't find the container with id db0739ad564d2223b458634f18979607df145900f04a94a03858b770287d85ba Sep 18 17:15:24.006993 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:24.006757 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qrbrr"] Sep 18 17:15:24.025559 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:24.025520 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k8tvl" event={"ID":"e62a1939-824b-49c5-857a-c0c41fb3c31a","Type":"ContainerStarted","Data":"db0739ad564d2223b458634f18979607df145900f04a94a03858b770287d85ba"} Sep 18 17:15:24.101946 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:15:24.101907 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf831002_a9cc_4239_9821_98618e18e764.slice/crio-9ecdfd8500d26dc49e3de3419ec03aaf9313e3ee4ba82861a0fef2d44aba3095 WatchSource:0}: Error finding container 9ecdfd8500d26dc49e3de3419ec03aaf9313e3ee4ba82861a0fef2d44aba3095: Status 404 returned error can't find the container with id 9ecdfd8500d26dc49e3de3419ec03aaf9313e3ee4ba82861a0fef2d44aba3095 Sep 18 17:15:25.030019 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:25.029982 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qrbrr" event={"ID":"af831002-a9cc-4239-9821-98618e18e764","Type":"ContainerStarted","Data":"9ecdfd8500d26dc49e3de3419ec03aaf9313e3ee4ba82861a0fef2d44aba3095"} Sep 18 17:15:25.034257 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:25.034223 2560 generic.go:356] "Generic (PLEG): container finished" podID="492a5d0b-c698-4eab-b0d6-5c2961725674" containerID="ada877d43ab37ea3fcb0f6b19c974661144c7c59dde05b6ee89b95ffe83d3cfc" exitCode=0 Sep 18 17:15:25.034436 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:25.034329 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7ns6" event={"ID":"492a5d0b-c698-4eab-b0d6-5c2961725674","Type":"ContainerDied","Data":"ada877d43ab37ea3fcb0f6b19c974661144c7c59dde05b6ee89b95ffe83d3cfc"} Sep 18 17:15:25.448864 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:25.448818 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:25.449094 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:25.449075 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 17:15:25.449178 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:25.449144 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls podName:ddf535f2-88b7-40b0-8d49-c9bfe97b3679 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:29.449128837 +0000 UTC m=+42.702289633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls") pod "dns-default-qqglr" (UID: "ddf535f2-88b7-40b0-8d49-c9bfe97b3679") : secret "dns-default-metrics-tls" not found Sep 18 17:15:26.040430 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:26.040392 2560 generic.go:356] "Generic (PLEG): container finished" podID="492a5d0b-c698-4eab-b0d6-5c2961725674" containerID="8f0f608684a080eb922b3b20e356ed455a7226584f8205f15e8ad5297d603cec" exitCode=0 Sep 18 17:15:26.040836 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:26.040455 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7ns6" event={"ID":"492a5d0b-c698-4eab-b0d6-5c2961725674","Type":"ContainerDied","Data":"8f0f608684a080eb922b3b20e356ed455a7226584f8205f15e8ad5297d603cec"} Sep 18 17:15:26.594715 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:26.594607 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:26.693589 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:26.693523 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:27.044788 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:27.044756 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k8tvl" event={"ID":"e62a1939-824b-49c5-857a-c0c41fb3c31a","Type":"ContainerStarted","Data":"f3a181dd2059807d6a00956b0e982d641bce9f72d46f777dcb7c9b23e0428d96"} Sep 18 17:15:27.048904 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:27.048870 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x7ns6" event={"ID":"492a5d0b-c698-4eab-b0d6-5c2961725674","Type":"ContainerStarted","Data":"b0d43a4987d373bd0a72d9d5a74eb170f43f62af0e21ca92803210920b5c7e4b"} Sep 18 17:15:27.064991 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:27.064926 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k8tvl" podStartSLOduration=3.456855427 podStartE2EDuration="6.064905672s" totalImagesPullingTime="2.608050245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:27.064439838 +0000 UTC m=+40.317600687" watchObservedRunningTime="2026-09-18 17:15:27.064905672 +0000 UTC m=+40.318066491" Sep 18 17:15:27.087621 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:27.087573 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x7ns6" podStartSLOduration=9.792370553 podStartE2EDuration="39.087555099s" totalImagesPullingTime="29.295184546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:27.085837813 +0000 UTC m=+40.338998649" watchObservedRunningTime="2026-09-18 17:15:27.087555099 +0000 UTC m=+40.340715913" Sep 18 17:15:27.166684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:27.166653 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:27.267354 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:27.267284 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:27.993484 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:27.993446 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:28.094514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:28.094451 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:28.217854 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:28.217818 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:28.536502 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:28.536471 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:28.630330 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:28.630234 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:29.055198 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:29.055167 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qrbrr" event={"ID":"af831002-a9cc-4239-9821-98618e18e764","Type":"ContainerStarted","Data":"a7a10d581fb157eb8cbc4c6ec26bb85b46f8ced2629c5bfd4a4127a8f8c780d3"} Sep 18 17:15:29.072238 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:29.072197 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qrbrr" podStartSLOduration=35.889320919 podStartE2EDuration="40.07218457s" totalImagesPullingTime="4.182863651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:29.071960023 +0000 UTC m=+42.325120862" watchObservedRunningTime="2026-09-18 17:15:29.07218457 +0000 UTC m=+42.325345386" Sep 18 17:15:29.478044 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:29.478003 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:29.478412 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:29.478135 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 17:15:29.478412 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:29.478184 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls podName:ddf535f2-88b7-40b0-8d49-c9bfe97b3679 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:37.478172197 +0000 UTC m=+50.731332992 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls") pod "dns-default-qqglr" (UID: "ddf535f2-88b7-40b0-8d49-c9bfe97b3679") : secret "dns-default-metrics-tls" not found Sep 18 17:15:34.491362 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.491325 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf"] Sep 18 17:15:34.496521 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.496504 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:34.500741 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.500710 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 18 17:15:34.502015 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.501992 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 18 17:15:34.503105 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.503081 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 18 17:15:34.503213 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.503189 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 18 17:15:34.503825 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.503800 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8zkxl\"" Sep 18 17:15:34.508373 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.508349 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf"] Sep 18 17:15:34.600905 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.600876 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-dp86w"] Sep 18 17:15:34.604176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.604161 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:34.606935 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.606914 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 18 17:15:34.607060 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.606937 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q6w5q\"" Sep 18 17:15:34.607534 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.607511 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 18 17:15:34.607635 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.607519 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 18 17:15:34.607687 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.607648 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 18 17:15:34.610507 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.610469 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:34.610619 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.610596 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a4216580-a12b-49e1-b2e1-607171a27a96-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:34.610670 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.610654 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djbgl\" (UniqueName: \"kubernetes.io/projected/a4216580-a12b-49e1-b2e1-607171a27a96-kube-api-access-djbgl\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:34.612519 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.612503 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 18 17:15:34.617805 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.617785 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-dp86w"] Sep 18 17:15:34.692179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.692150 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:34.711465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.711437 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5aa5aa57-bbfa-461a-b12a-e32ad883c338-trusted-ca\") pod \"console-operator-fd4bb46c-dp86w\" (UID: \"5aa5aa57-bbfa-461a-b12a-e32ad883c338\") " pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:34.711616 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.711479 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xlfd\" (UniqueName: \"kubernetes.io/projected/5aa5aa57-bbfa-461a-b12a-e32ad883c338-kube-api-access-5xlfd\") pod \"console-operator-fd4bb46c-dp86w\" (UID: \"5aa5aa57-bbfa-461a-b12a-e32ad883c338\") " pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:34.711616 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.711505 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djbgl\" (UniqueName: \"kubernetes.io/projected/a4216580-a12b-49e1-b2e1-607171a27a96-kube-api-access-djbgl\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:34.711616 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.711570 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5aa5aa57-bbfa-461a-b12a-e32ad883c338-serving-cert\") pod \"console-operator-fd4bb46c-dp86w\" (UID: \"5aa5aa57-bbfa-461a-b12a-e32ad883c338\") " pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:34.711616 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.711593 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5aa5aa57-bbfa-461a-b12a-e32ad883c338-config\") pod \"console-operator-fd4bb46c-dp86w\" (UID: \"5aa5aa57-bbfa-461a-b12a-e32ad883c338\") " pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:34.711763 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.711638 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:34.711763 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.711706 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a4216580-a12b-49e1-b2e1-607171a27a96-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:34.711763 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:34.711725 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 17:15:34.711851 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:34.711780 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls podName:a4216580-a12b-49e1-b2e1-607171a27a96 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:35.211763887 +0000 UTC m=+48.464924682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4xbwf" (UID: "a4216580-a12b-49e1-b2e1-607171a27a96") : secret "cluster-monitoring-operator-tls" not found Sep 18 17:15:34.712365 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.712349 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a4216580-a12b-49e1-b2e1-607171a27a96-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:34.727196 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.727174 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djbgl\" (UniqueName: \"kubernetes.io/projected/a4216580-a12b-49e1-b2e1-607171a27a96-kube-api-access-djbgl\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:34.763586 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.763487 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:34.812772 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.812742 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5aa5aa57-bbfa-461a-b12a-e32ad883c338-trusted-ca\") pod \"console-operator-fd4bb46c-dp86w\" (UID: \"5aa5aa57-bbfa-461a-b12a-e32ad883c338\") " pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:34.812927 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.812782 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xlfd\" (UniqueName: \"kubernetes.io/projected/5aa5aa57-bbfa-461a-b12a-e32ad883c338-kube-api-access-5xlfd\") pod \"console-operator-fd4bb46c-dp86w\" (UID: \"5aa5aa57-bbfa-461a-b12a-e32ad883c338\") " pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:34.812927 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.812817 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5aa5aa57-bbfa-461a-b12a-e32ad883c338-serving-cert\") pod \"console-operator-fd4bb46c-dp86w\" (UID: \"5aa5aa57-bbfa-461a-b12a-e32ad883c338\") " pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:34.812927 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.812842 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5aa5aa57-bbfa-461a-b12a-e32ad883c338-config\") pod \"console-operator-fd4bb46c-dp86w\" (UID: \"5aa5aa57-bbfa-461a-b12a-e32ad883c338\") " pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:34.813557 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.813532 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5aa5aa57-bbfa-461a-b12a-e32ad883c338-config\") pod \"console-operator-fd4bb46c-dp86w\" (UID: \"5aa5aa57-bbfa-461a-b12a-e32ad883c338\") " pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:34.813664 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.813634 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5aa5aa57-bbfa-461a-b12a-e32ad883c338-trusted-ca\") pod \"console-operator-fd4bb46c-dp86w\" (UID: \"5aa5aa57-bbfa-461a-b12a-e32ad883c338\") " pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:34.815191 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.815166 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5aa5aa57-bbfa-461a-b12a-e32ad883c338-serving-cert\") pod \"console-operator-fd4bb46c-dp86w\" (UID: \"5aa5aa57-bbfa-461a-b12a-e32ad883c338\") " pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:34.821452 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.821429 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xlfd\" (UniqueName: \"kubernetes.io/projected/5aa5aa57-bbfa-461a-b12a-e32ad883c338-kube-api-access-5xlfd\") pod \"console-operator-fd4bb46c-dp86w\" (UID: \"5aa5aa57-bbfa-461a-b12a-e32ad883c338\") " pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:34.876945 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.876917 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d6dc57758-jwp4d"] Sep 18 17:15:34.886963 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.886941 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:34.890574 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.890549 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 18 17:15:34.890686 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.890579 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 18 17:15:34.890686 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.890616 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 18 17:15:34.890686 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.890622 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mr4mj\"" Sep 18 17:15:34.897574 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.897554 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d6dc57758-jwp4d"] Sep 18 17:15:34.900555 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.900532 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 18 17:15:34.913597 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:34.913575 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:35.014826 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.014750 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acaef214-5866-40a6-bb5e-dce035b34f5b-installation-pull-secrets\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.014826 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.014789 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acaef214-5866-40a6-bb5e-dce035b34f5b-ca-trust-extracted\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.014826 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.014815 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2dsm\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-kube-api-access-p2dsm\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.015013 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.014833 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-certificates\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.015013 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.014854 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.015013 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.014895 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-bound-sa-token\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.015013 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.014946 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acaef214-5866-40a6-bb5e-dce035b34f5b-image-registry-private-configuration\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.015013 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.014975 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acaef214-5866-40a6-bb5e-dce035b34f5b-trusted-ca\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.115894 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.115866 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acaef214-5866-40a6-bb5e-dce035b34f5b-installation-pull-secrets\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.115894 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.115897 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acaef214-5866-40a6-bb5e-dce035b34f5b-ca-trust-extracted\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.116113 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.115927 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2dsm\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-kube-api-access-p2dsm\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.116113 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.116100 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-certificates\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.116219 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.116121 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.116219 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.116148 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-bound-sa-token\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.116219 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.116176 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acaef214-5866-40a6-bb5e-dce035b34f5b-image-registry-private-configuration\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.116219 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.116212 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acaef214-5866-40a6-bb5e-dce035b34f5b-trusted-ca\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.116437 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:35.116263 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 17:15:35.116437 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:35.116282 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d6dc57758-jwp4d: secret "image-registry-tls" not found Sep 18 17:15:35.116437 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.116314 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acaef214-5866-40a6-bb5e-dce035b34f5b-ca-trust-extracted\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.116437 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:35.116371 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls podName:acaef214-5866-40a6-bb5e-dce035b34f5b nodeName:}" failed. No retries permitted until 2026-09-18 17:15:35.616348973 +0000 UTC m=+48.869509783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls") pod "image-registry-d6dc57758-jwp4d" (UID: "acaef214-5866-40a6-bb5e-dce035b34f5b") : secret "image-registry-tls" not found Sep 18 17:15:35.116680 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.116658 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-certificates\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.117066 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.117049 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acaef214-5866-40a6-bb5e-dce035b34f5b-trusted-ca\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.118436 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.118415 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acaef214-5866-40a6-bb5e-dce035b34f5b-installation-pull-secrets\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.118505 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.118482 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acaef214-5866-40a6-bb5e-dce035b34f5b-image-registry-private-configuration\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.127042 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.126728 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2dsm\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-kube-api-access-p2dsm\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.128942 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.128925 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-bound-sa-token\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.185853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.185831 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-dp86w"] Sep 18 17:15:35.188007 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:15:35.187983 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa5aa57_bbfa_461a_b12a_e32ad883c338.slice/crio-7d31ba8f4821881ea149ab937e753fcec17d5c27dd68d820f1024cf473fcd36b WatchSource:0}: Error finding container 7d31ba8f4821881ea149ab937e753fcec17d5c27dd68d820f1024cf473fcd36b: Status 404 returned error can't find the container with id 7d31ba8f4821881ea149ab937e753fcec17d5c27dd68d820f1024cf473fcd36b Sep 18 17:15:35.217133 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.217105 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:35.217228 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:35.217217 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 17:15:35.217282 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:35.217274 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls podName:a4216580-a12b-49e1-b2e1-607171a27a96 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:36.217261189 +0000 UTC m=+49.470421997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4xbwf" (UID: "a4216580-a12b-49e1-b2e1-607171a27a96") : secret "cluster-monitoring-operator-tls" not found Sep 18 17:15:35.619629 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:35.619593 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:35.620000 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:35.619715 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 17:15:35.620000 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:35.619728 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d6dc57758-jwp4d: secret "image-registry-tls" not found Sep 18 17:15:35.620000 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:35.619786 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls podName:acaef214-5866-40a6-bb5e-dce035b34f5b nodeName:}" failed. No retries permitted until 2026-09-18 17:15:36.61977279 +0000 UTC m=+49.872933598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls") pod "image-registry-d6dc57758-jwp4d" (UID: "acaef214-5866-40a6-bb5e-dce035b34f5b") : secret "image-registry-tls" not found Sep 18 17:15:36.073145 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.073111 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" event={"ID":"5aa5aa57-bbfa-461a-b12a-e32ad883c338","Type":"ContainerStarted","Data":"7d31ba8f4821881ea149ab937e753fcec17d5c27dd68d820f1024cf473fcd36b"} Sep 18 17:15:36.224156 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.224116 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:36.224346 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:36.224324 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 17:15:36.224416 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:36.224405 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls podName:a4216580-a12b-49e1-b2e1-607171a27a96 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:38.22438471 +0000 UTC m=+51.477545519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4xbwf" (UID: "a4216580-a12b-49e1-b2e1-607171a27a96") : secret "cluster-monitoring-operator-tls" not found Sep 18 17:15:36.428254 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.428221 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4"] Sep 18 17:15:36.464869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.464844 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4"] Sep 18 17:15:36.465004 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.464925 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" Sep 18 17:15:36.468585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.468560 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 18 17:15:36.468703 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.468585 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 18 17:15:36.468703 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.468560 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 18 17:15:36.469667 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.469651 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-z42d5\"" Sep 18 17:15:36.469667 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.469659 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 18 17:15:36.627855 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.627826 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/156a0989-b40d-4b6b-84cf-c8afb86fe65c-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-pbzv4\" (UID: \"156a0989-b40d-4b6b-84cf-c8afb86fe65c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" Sep 18 17:15:36.628230 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.627867 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/156a0989-b40d-4b6b-84cf-c8afb86fe65c-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-pbzv4\" (UID: \"156a0989-b40d-4b6b-84cf-c8afb86fe65c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" Sep 18 17:15:36.628230 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.627922 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmrpc\" (UniqueName: \"kubernetes.io/projected/156a0989-b40d-4b6b-84cf-c8afb86fe65c-kube-api-access-xmrpc\") pod \"kube-storage-version-migrator-operator-55889f7b47-pbzv4\" (UID: \"156a0989-b40d-4b6b-84cf-c8afb86fe65c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" Sep 18 17:15:36.628230 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.627964 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:36.628230 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:36.628043 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 17:15:36.628230 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:36.628053 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d6dc57758-jwp4d: secret "image-registry-tls" not found Sep 18 17:15:36.628230 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:36.628094 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls podName:acaef214-5866-40a6-bb5e-dce035b34f5b nodeName:}" failed. No retries permitted until 2026-09-18 17:15:38.628079926 +0000 UTC m=+51.881240721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls") pod "image-registry-d6dc57758-jwp4d" (UID: "acaef214-5866-40a6-bb5e-dce035b34f5b") : secret "image-registry-tls" not found Sep 18 17:15:36.728617 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.728537 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/156a0989-b40d-4b6b-84cf-c8afb86fe65c-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-pbzv4\" (UID: \"156a0989-b40d-4b6b-84cf-c8afb86fe65c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" Sep 18 17:15:36.728617 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.728585 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/156a0989-b40d-4b6b-84cf-c8afb86fe65c-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-pbzv4\" (UID: \"156a0989-b40d-4b6b-84cf-c8afb86fe65c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" Sep 18 17:15:36.728617 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.728615 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrpc\" (UniqueName: \"kubernetes.io/projected/156a0989-b40d-4b6b-84cf-c8afb86fe65c-kube-api-access-xmrpc\") pod \"kube-storage-version-migrator-operator-55889f7b47-pbzv4\" (UID: \"156a0989-b40d-4b6b-84cf-c8afb86fe65c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" Sep 18 17:15:36.729255 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.729230 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/156a0989-b40d-4b6b-84cf-c8afb86fe65c-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-pbzv4\" (UID: \"156a0989-b40d-4b6b-84cf-c8afb86fe65c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" Sep 18 17:15:36.730899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.730878 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/156a0989-b40d-4b6b-84cf-c8afb86fe65c-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-pbzv4\" (UID: \"156a0989-b40d-4b6b-84cf-c8afb86fe65c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" Sep 18 17:15:36.737649 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.737629 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmrpc\" (UniqueName: \"kubernetes.io/projected/156a0989-b40d-4b6b-84cf-c8afb86fe65c-kube-api-access-xmrpc\") pod \"kube-storage-version-migrator-operator-55889f7b47-pbzv4\" (UID: \"156a0989-b40d-4b6b-84cf-c8afb86fe65c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" Sep 18 17:15:36.774191 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.774170 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" Sep 18 17:15:36.960837 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:36.960809 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:37.025053 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:37.024989 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:37.470157 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:37.470133 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4"] Sep 18 17:15:37.520173 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:15:37.520148 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156a0989_b40d_4b6b_84cf_c8afb86fe65c.slice/crio-714ab7f7368f5e9940cb6628d4008413f5d1cebe57fc7b2729562b4ad171ba9f WatchSource:0}: Error finding container 714ab7f7368f5e9940cb6628d4008413f5d1cebe57fc7b2729562b4ad171ba9f: Status 404 returned error can't find the container with id 714ab7f7368f5e9940cb6628d4008413f5d1cebe57fc7b2729562b4ad171ba9f Sep 18 17:15:37.536026 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:37.536001 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:37.536163 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:37.536146 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 18 17:15:37.536212 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:37.536206 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls podName:ddf535f2-88b7-40b0-8d49-c9bfe97b3679 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:53.536189697 +0000 UTC m=+66.789350492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls") pod "dns-default-qqglr" (UID: "ddf535f2-88b7-40b0-8d49-c9bfe97b3679") : secret "dns-default-metrics-tls" not found Sep 18 17:15:38.080876 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:38.080850 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/0.log" Sep 18 17:15:38.081307 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:38.080888 2560 generic.go:356] "Generic (PLEG): container finished" podID="5aa5aa57-bbfa-461a-b12a-e32ad883c338" containerID="d4a88e38db3818ee1f5fcb166e3fb46db3128fed4a011648a2d9ecbfdf2afe50" exitCode=255 Sep 18 17:15:38.081307 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:38.080955 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" event={"ID":"5aa5aa57-bbfa-461a-b12a-e32ad883c338","Type":"ContainerDied","Data":"d4a88e38db3818ee1f5fcb166e3fb46db3128fed4a011648a2d9ecbfdf2afe50"} Sep 18 17:15:38.081307 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:38.081189 2560 scope.go:117] "RemoveContainer" containerID="d4a88e38db3818ee1f5fcb166e3fb46db3128fed4a011648a2d9ecbfdf2afe50" Sep 18 17:15:38.082317 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:38.082276 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" event={"ID":"156a0989-b40d-4b6b-84cf-c8afb86fe65c","Type":"ContainerStarted","Data":"714ab7f7368f5e9940cb6628d4008413f5d1cebe57fc7b2729562b4ad171ba9f"} Sep 18 17:15:38.241803 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:38.241771 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:38.241927 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:38.241883 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 17:15:38.241985 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:38.241946 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls podName:a4216580-a12b-49e1-b2e1-607171a27a96 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:42.241928638 +0000 UTC m=+55.495089450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4xbwf" (UID: "a4216580-a12b-49e1-b2e1-607171a27a96") : secret "cluster-monitoring-operator-tls" not found Sep 18 17:15:38.644627 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:38.644599 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:38.644804 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:38.644707 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 17:15:38.644804 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:38.644720 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d6dc57758-jwp4d: secret "image-registry-tls" not found Sep 18 17:15:38.644804 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:38.644772 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls podName:acaef214-5866-40a6-bb5e-dce035b34f5b nodeName:}" failed. No retries permitted until 2026-09-18 17:15:42.644755001 +0000 UTC m=+55.897915800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls") pod "image-registry-d6dc57758-jwp4d" (UID: "acaef214-5866-40a6-bb5e-dce035b34f5b") : secret "image-registry-tls" not found Sep 18 17:15:38.994415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:38.994380 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:39.024691 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.024664 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s9kbh"] Sep 18 17:15:39.047229 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.047204 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.050485 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.050454 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 18 17:15:39.051661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.051638 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2sl55\"" Sep 18 17:15:39.051818 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.051696 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 18 17:15:39.051818 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.051639 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 18 17:15:39.052039 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.052006 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 18 17:15:39.053326 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.053302 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s9kbh"] Sep 18 17:15:39.085557 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.085536 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:39.085850 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.085666 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:15:39.086100 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.086084 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/0.log" Sep 18 17:15:39.086178 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.086123 2560 generic.go:356] "Generic (PLEG): container finished" podID="5aa5aa57-bbfa-461a-b12a-e32ad883c338" containerID="5a29ffa870479a1326b123cf2f84e4700acbbcead141267aa92bbb1c4f5beacd" exitCode=255 Sep 18 17:15:39.086230 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.086189 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" event={"ID":"5aa5aa57-bbfa-461a-b12a-e32ad883c338","Type":"ContainerDied","Data":"5a29ffa870479a1326b123cf2f84e4700acbbcead141267aa92bbb1c4f5beacd"} Sep 18 17:15:39.086278 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.086253 2560 scope.go:117] "RemoveContainer" containerID="d4a88e38db3818ee1f5fcb166e3fb46db3128fed4a011648a2d9ecbfdf2afe50" Sep 18 17:15:39.086501 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.086485 2560 scope.go:117] "RemoveContainer" containerID="5a29ffa870479a1326b123cf2f84e4700acbbcead141267aa92bbb1c4f5beacd" Sep 18 17:15:39.086668 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:39.086648 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-dp86w_openshift-console-operator(5aa5aa57-bbfa-461a-b12a-e32ad883c338)\"" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" podUID="5aa5aa57-bbfa-461a-b12a-e32ad883c338" Sep 18 17:15:39.148593 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.148573 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.148692 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.148603 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.148692 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.148625 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c6aba87-34e8-41a8-9157-96b93bd9dbda-data-volume\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.148768 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.148744 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c6aba87-34e8-41a8-9157-96b93bd9dbda-crio-socket\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.148801 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.148787 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzp7t\" (UniqueName: \"kubernetes.io/projected/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-api-access-wzp7t\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.249123 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.249064 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c6aba87-34e8-41a8-9157-96b93bd9dbda-crio-socket\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.249123 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.249123 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzp7t\" (UniqueName: \"kubernetes.io/projected/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-api-access-wzp7t\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.249259 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.249172 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.249259 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.249196 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.249259 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.249220 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c6aba87-34e8-41a8-9157-96b93bd9dbda-data-volume\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.249444 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:39.249276 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 17:15:39.249444 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:39.249366 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm podName:6c6aba87-34e8-41a8-9157-96b93bd9dbda nodeName:}" failed. No retries permitted until 2026-09-18 17:15:39.749347241 +0000 UTC m=+53.002508049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s9kbh" (UID: "6c6aba87-34e8-41a8-9157-96b93bd9dbda") : configmap "kube-rbac-proxy" not found Sep 18 17:15:39.249444 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:39.249368 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 17:15:39.249444 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:39.249426 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls podName:6c6aba87-34e8-41a8-9157-96b93bd9dbda nodeName:}" failed. No retries permitted until 2026-09-18 17:15:39.749409856 +0000 UTC m=+53.002570663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s9kbh" (UID: "6c6aba87-34e8-41a8-9157-96b93bd9dbda") : secret "insights-runtime-extractor-tls" not found Sep 18 17:15:39.249444 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.249278 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c6aba87-34e8-41a8-9157-96b93bd9dbda-crio-socket\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.249690 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.249584 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c6aba87-34e8-41a8-9157-96b93bd9dbda-data-volume\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.257975 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.257956 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzp7t\" (UniqueName: \"kubernetes.io/projected/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-api-access-wzp7t\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.312399 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.312366 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-8pjgd"] Sep 18 17:15:39.321530 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.321512 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8pjgd" Sep 18 17:15:39.324365 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.324346 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-plm2w\"" Sep 18 17:15:39.327001 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.326945 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-8pjgd"] Sep 18 17:15:39.451207 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.451174 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54grl\" (UniqueName: \"kubernetes.io/projected/21e66daa-fd3d-4105-9900-119c74f568d7-kube-api-access-54grl\") pod \"network-check-source-7cbc45fddc-8pjgd\" (UID: \"21e66daa-fd3d-4105-9900-119c74f568d7\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8pjgd" Sep 18 17:15:39.552625 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.552535 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54grl\" (UniqueName: \"kubernetes.io/projected/21e66daa-fd3d-4105-9900-119c74f568d7-kube-api-access-54grl\") pod \"network-check-source-7cbc45fddc-8pjgd\" (UID: \"21e66daa-fd3d-4105-9900-119c74f568d7\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8pjgd" Sep 18 17:15:39.563054 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.563026 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54grl\" (UniqueName: \"kubernetes.io/projected/21e66daa-fd3d-4105-9900-119c74f568d7-kube-api-access-54grl\") pod \"network-check-source-7cbc45fddc-8pjgd\" (UID: \"21e66daa-fd3d-4105-9900-119c74f568d7\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8pjgd" Sep 18 17:15:39.631978 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.631944 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8pjgd" Sep 18 17:15:39.754750 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.754709 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.754946 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.754760 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:39.754946 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:39.754876 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 17:15:39.754946 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:39.754935 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls podName:6c6aba87-34e8-41a8-9157-96b93bd9dbda nodeName:}" failed. No retries permitted until 2026-09-18 17:15:40.754920811 +0000 UTC m=+54.008081606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s9kbh" (UID: "6c6aba87-34e8-41a8-9157-96b93bd9dbda") : secret "insights-runtime-extractor-tls" not found Sep 18 17:15:39.755113 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:39.754951 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 17:15:39.755113 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:39.755052 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm podName:6c6aba87-34e8-41a8-9157-96b93bd9dbda nodeName:}" failed. No retries permitted until 2026-09-18 17:15:40.755033474 +0000 UTC m=+54.008194276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s9kbh" (UID: "6c6aba87-34e8-41a8-9157-96b93bd9dbda") : configmap "kube-rbac-proxy" not found Sep 18 17:15:39.979397 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:39.979322 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-8pjgd"] Sep 18 17:15:39.981227 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:15:39.981202 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e66daa_fd3d_4105_9900_119c74f568d7.slice/crio-86c3a4f3f3a9a74adb84c74b651cdc853a0965bfbab3cb78a1a3902f7178c471 WatchSource:0}: Error finding container 86c3a4f3f3a9a74adb84c74b651cdc853a0965bfbab3cb78a1a3902f7178c471: Status 404 returned error can't find the container with id 86c3a4f3f3a9a74adb84c74b651cdc853a0965bfbab3cb78a1a3902f7178c471 Sep 18 17:15:40.094178 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:40.094155 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:15:40.094594 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:40.094484 2560 scope.go:117] "RemoveContainer" containerID="5a29ffa870479a1326b123cf2f84e4700acbbcead141267aa92bbb1c4f5beacd" Sep 18 17:15:40.094656 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:40.094634 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-dp86w_openshift-console-operator(5aa5aa57-bbfa-461a-b12a-e32ad883c338)\"" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" podUID="5aa5aa57-bbfa-461a-b12a-e32ad883c338" Sep 18 17:15:40.095917 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:40.095895 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" event={"ID":"156a0989-b40d-4b6b-84cf-c8afb86fe65c","Type":"ContainerStarted","Data":"6fea6feb612b959102fccdc5677a301cfaed3e48ded35b1a72308f8ef406b3eb"} Sep 18 17:15:40.097104 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:40.097085 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8pjgd" event={"ID":"21e66daa-fd3d-4105-9900-119c74f568d7","Type":"ContainerStarted","Data":"86c3a4f3f3a9a74adb84c74b651cdc853a0965bfbab3cb78a1a3902f7178c471"} Sep 18 17:15:40.127782 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:40.127742 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" podStartSLOduration=1.701384913 podStartE2EDuration="4.127730382s" totalImagesPullingTime="2.426345469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:40.126984536 +0000 UTC m=+53.380145381" watchObservedRunningTime="2026-09-18 17:15:40.127730382 +0000 UTC m=+53.380891199" Sep 18 17:15:40.708287 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:40.708262 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4vxw9_4a44888c-a9bf-4378-8785-26bc77ef43bb/dns-node-resolver/0.log" Sep 18 17:15:40.762760 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:40.762728 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:40.762942 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:40.762774 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:40.762942 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:40.762856 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 17:15:40.762942 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:40.762934 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm podName:6c6aba87-34e8-41a8-9157-96b93bd9dbda nodeName:}" failed. No retries permitted until 2026-09-18 17:15:42.762913918 +0000 UTC m=+56.016074729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s9kbh" (UID: "6c6aba87-34e8-41a8-9157-96b93bd9dbda") : configmap "kube-rbac-proxy" not found Sep 18 17:15:40.763108 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:40.762958 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 17:15:40.763108 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:40.763022 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls podName:6c6aba87-34e8-41a8-9157-96b93bd9dbda nodeName:}" failed. No retries permitted until 2026-09-18 17:15:42.763004681 +0000 UTC m=+56.016165492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s9kbh" (UID: "6c6aba87-34e8-41a8-9157-96b93bd9dbda") : secret "insights-runtime-extractor-tls" not found Sep 18 17:15:41.306000 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:41.305961 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:41.418830 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:41.418794 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:41.506581 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:41.506540 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:41.910284 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:41.910257 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-44qw2_b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73/node-ca/0.log" Sep 18 17:15:42.276330 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:42.276226 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:42.276492 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:42.276368 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 17:15:42.276492 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:42.276432 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls podName:a4216580-a12b-49e1-b2e1-607171a27a96 nodeName:}" failed. No retries permitted until 2026-09-18 17:15:50.276412204 +0000 UTC m=+63.529573003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4xbwf" (UID: "a4216580-a12b-49e1-b2e1-607171a27a96") : secret "cluster-monitoring-operator-tls" not found Sep 18 17:15:42.526070 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:42.526037 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:42.595473 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:42.595371 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:42.679016 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:42.678973 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:42.679212 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:42.679149 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 17:15:42.679212 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:42.679167 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d6dc57758-jwp4d: secret "image-registry-tls" not found Sep 18 17:15:42.679351 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:42.679236 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls podName:acaef214-5866-40a6-bb5e-dce035b34f5b nodeName:}" failed. No retries permitted until 2026-09-18 17:15:50.67921358 +0000 UTC m=+63.932374381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls") pod "image-registry-d6dc57758-jwp4d" (UID: "acaef214-5866-40a6-bb5e-dce035b34f5b") : secret "image-registry-tls" not found Sep 18 17:15:42.709547 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:42.709518 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k8tvl_e62a1939-824b-49c5-857a-c0c41fb3c31a/serve-healthcheck-canary/0.log" Sep 18 17:15:42.779629 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:42.779590 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:42.779827 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:42.779645 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:42.779827 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:42.779724 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 17:15:42.779827 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:42.779814 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm podName:6c6aba87-34e8-41a8-9157-96b93bd9dbda nodeName:}" failed. No retries permitted until 2026-09-18 17:15:46.779790745 +0000 UTC m=+60.032951549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s9kbh" (UID: "6c6aba87-34e8-41a8-9157-96b93bd9dbda") : configmap "kube-rbac-proxy" not found Sep 18 17:15:42.779997 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:42.779857 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 17:15:42.779997 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:42.779950 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls podName:6c6aba87-34e8-41a8-9157-96b93bd9dbda nodeName:}" failed. No retries permitted until 2026-09-18 17:15:46.779918237 +0000 UTC m=+60.033079084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s9kbh" (UID: "6c6aba87-34e8-41a8-9157-96b93bd9dbda") : secret "insights-runtime-extractor-tls" not found Sep 18 17:15:42.968620 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:42.968588 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:43.038129 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:43.038100 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:43.229684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:43.229611 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:44.112106 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:44.112070 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8pjgd" event={"ID":"21e66daa-fd3d-4105-9900-119c74f568d7","Type":"ContainerStarted","Data":"64778070d344c87e6c37e2c9aa7ad4cb5614815591d9c59dc2af528b8a979633"} Sep 18 17:15:44.128840 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:44.128786 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-8pjgd" podStartSLOduration=1.7745645730000001 podStartE2EDuration="5.128770184s" totalImagesPullingTime="3.354205611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:44.127606908 +0000 UTC m=+57.380767744" watchObservedRunningTime="2026-09-18 17:15:44.128770184 +0000 UTC m=+57.381931001" Sep 18 17:15:44.379792 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:44.379714 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:44.866790 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:44.866744 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:44.913827 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:44.913792 2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:44.913987 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:44.913853 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:15:44.914172 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:44.914160 2560 scope.go:117] "RemoveContainer" containerID="5a29ffa870479a1326b123cf2f84e4700acbbcead141267aa92bbb1c4f5beacd" Sep 18 17:15:44.914356 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:44.914338 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-dp86w_openshift-console-operator(5aa5aa57-bbfa-461a-b12a-e32ad883c338)\"" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" podUID="5aa5aa57-bbfa-461a-b12a-e32ad883c338" Sep 18 17:15:44.953883 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:44.953850 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:45.174350 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:45.174320 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:45.238089 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:45.238062 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:45.871243 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:45.871208 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:46.815565 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:46.815525 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:46.815565 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:46.815564 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:46.815978 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:46.815650 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 18 17:15:46.815978 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:46.815676 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 17:15:46.815978 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:46.815731 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm podName:6c6aba87-34e8-41a8-9157-96b93bd9dbda nodeName:}" failed. No retries permitted until 2026-09-18 17:15:54.815714115 +0000 UTC m=+68.068874915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s9kbh" (UID: "6c6aba87-34e8-41a8-9157-96b93bd9dbda") : configmap "kube-rbac-proxy" not found Sep 18 17:15:46.815978 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:46.815747 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls podName:6c6aba87-34e8-41a8-9157-96b93bd9dbda nodeName:}" failed. No retries permitted until 2026-09-18 17:15:54.815741346 +0000 UTC m=+68.068902141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s9kbh" (UID: "6c6aba87-34e8-41a8-9157-96b93bd9dbda") : secret "insights-runtime-extractor-tls" not found Sep 18 17:15:48.023111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:48.023084 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qf4rc" Sep 18 17:15:48.158807 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:48.158775 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:48.307003 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:48.306923 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:48.423970 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:48.423940 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:48.517914 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:48.517886 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:48.632364 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:48.631620 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:48.866254 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:48.866214 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:49.594590 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:49.594546 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:49.695360 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:49.695333 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:49.857828 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:49.857763 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:50.342832 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:50.342795 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:15:50.342997 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:50.342934 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 18 17:15:50.343038 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:50.342999 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls podName:a4216580-a12b-49e1-b2e1-607171a27a96 nodeName:}" failed. No retries permitted until 2026-09-18 17:16:06.34298223 +0000 UTC m=+79.596143024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4xbwf" (UID: "a4216580-a12b-49e1-b2e1-607171a27a96") : secret "cluster-monitoring-operator-tls" not found Sep 18 17:15:50.458973 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:50.458935 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:50.746127 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:50.746095 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls\") pod \"image-registry-d6dc57758-jwp4d\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:15:50.746509 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:50.746256 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 18 17:15:50.746509 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:50.746277 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d6dc57758-jwp4d: secret "image-registry-tls" not found Sep 18 17:15:50.746509 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:50.746355 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls podName:acaef214-5866-40a6-bb5e-dce035b34f5b nodeName:}" failed. No retries permitted until 2026-09-18 17:16:06.746337466 +0000 UTC m=+79.999498276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls") pod "image-registry-d6dc57758-jwp4d" (UID: "acaef214-5866-40a6-bb5e-dce035b34f5b") : secret "image-registry-tls" not found Sep 18 17:15:50.860286 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:50.860249 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:51.258173 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:51.258143 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:51.459931 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:51.459900 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:51.871961 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:51.871930 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:51.992179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:51.992151 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:52.093534 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:52.093511 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:52.193827 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:52.193807 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:52.462499 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:52.462430 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:52.863185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:52.863111 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:53.057342 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:53.057286 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:53.455390 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:53.455365 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:53.566262 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:53.566226 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:53.568553 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:53.568532 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddf535f2-88b7-40b0-8d49-c9bfe97b3679-metrics-tls\") pod \"dns-default-qqglr\" (UID: \"ddf535f2-88b7-40b0-8d49-c9bfe97b3679\") " pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:53.782870 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:53.782818 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99knh\"" Sep 18 17:15:53.790987 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:53.790961 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:54.056417 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.056353 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qqglr"] Sep 18 17:15:54.059651 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:15:54.059624 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf535f2_88b7_40b0_8d49_c9bfe97b3679.slice/crio-2540ae1eda5136b3038b0873f6c15eb8095f5ff27cafab9bc1ccc83adaf3d4aa WatchSource:0}: Error finding container 2540ae1eda5136b3038b0873f6c15eb8095f5ff27cafab9bc1ccc83adaf3d4aa: Status 404 returned error can't find the container with id 2540ae1eda5136b3038b0873f6c15eb8095f5ff27cafab9bc1ccc83adaf3d4aa Sep 18 17:15:54.061917 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.061895 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:54.139432 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.139410 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qqglr" event={"ID":"ddf535f2-88b7-40b0-8d49-c9bfe97b3679","Type":"ContainerStarted","Data":"2540ae1eda5136b3038b0873f6c15eb8095f5ff27cafab9bc1ccc83adaf3d4aa"} Sep 18 17:15:54.383177 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.383125 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:54.574674 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.574647 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:15:54.574837 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:54.574818 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 18 17:15:54.574908 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:54.574901 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs podName:cd4d7977-637e-4068-8d44-515534534834 nodeName:}" failed. No retries permitted until 2026-09-18 17:16:58.574882433 +0000 UTC m=+131.828043233 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs") pod "network-metrics-daemon-clj2d" (UID: "cd4d7977-637e-4068-8d44-515534534834") : secret "metrics-daemon-secret" not found Sep 18 17:15:54.666268 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.666245 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:54.675252 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.675227 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzxm\" (UniqueName: \"kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm\") pod \"network-check-target-wqx8x\" (UID: \"6883d049-759d-4fc5-af31-d30305f4b092\") " pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:54.678214 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.678194 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkzxm\" (UniqueName: \"kubernetes.io/projected/6883d049-759d-4fc5-af31-d30305f4b092-kube-api-access-dkzxm\") pod \"network-check-target-wqx8x\" (UID: \"6883d049-759d-4fc5-af31-d30305f4b092\") " pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:54.861936 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.861904 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:54.877213 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.877185 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:54.877383 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.877217 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:54.877383 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:54.877351 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 18 17:15:54.877507 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:15:54.877408 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls podName:6c6aba87-34e8-41a8-9157-96b93bd9dbda nodeName:}" failed. No retries permitted until 2026-09-18 17:16:10.877390812 +0000 UTC m=+84.130551608 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s9kbh" (UID: "6c6aba87-34e8-41a8-9157-96b93bd9dbda") : secret "insights-runtime-extractor-tls" not found Sep 18 17:15:54.877809 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.877786 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c6aba87-34e8-41a8-9157-96b93bd9dbda-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:15:54.973732 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.973657 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f2lgv\"" Sep 18 17:15:54.981361 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:54.981344 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:55.055537 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:55.055490 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:55.245161 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:55.245137 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wqx8x"] Sep 18 17:15:55.247387 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:15:55.247360 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6883d049_759d_4fc5_af31_d30305f4b092.slice/crio-0f88ffa5e8a013f1e27621882748b8e074a68c251bb47ec4c53cc9f7d1be8592 WatchSource:0}: Error finding container 0f88ffa5e8a013f1e27621882748b8e074a68c251bb47ec4c53cc9f7d1be8592: Status 404 returned error can't find the container with id 0f88ffa5e8a013f1e27621882748b8e074a68c251bb47ec4c53cc9f7d1be8592 Sep 18 17:15:55.463059 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:55.463025 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:56.146888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:56.146859 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qqglr" event={"ID":"ddf535f2-88b7-40b0-8d49-c9bfe97b3679","Type":"ContainerStarted","Data":"81a7365d92cc12a02f42d94e8c032d1b38ce1c7c36aecf953ec7fed5559c7888"} Sep 18 17:15:56.147051 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:56.146958 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qqglr" event={"ID":"ddf535f2-88b7-40b0-8d49-c9bfe97b3679","Type":"ContainerStarted","Data":"754db77d9203dadd30f47546d724f0855d036fbab25c4f7b064e37830df19df7"} Sep 18 17:15:56.147051 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:56.146993 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qqglr" Sep 18 17:15:56.148482 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:56.148457 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wqx8x" event={"ID":"6883d049-759d-4fc5-af31-d30305f4b092","Type":"ContainerStarted","Data":"ef66effc6ec81c1defffa990e81f6a52a17554d00ec9d90429f62a638aae61a0"} Sep 18 17:15:56.148585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:56.148493 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wqx8x" event={"ID":"6883d049-759d-4fc5-af31-d30305f4b092","Type":"ContainerStarted","Data":"0f88ffa5e8a013f1e27621882748b8e074a68c251bb47ec4c53cc9f7d1be8592"} Sep 18 17:15:56.148585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:56.148510 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:15:56.168391 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:56.168350 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qqglr" podStartSLOduration=33.3796896 podStartE2EDuration="35.168337645s" totalImagesPullingTime="1.788648045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:56.168169311 +0000 UTC m=+69.421330128" watchObservedRunningTime="2026-09-18 17:15:56.168337645 +0000 UTC m=+69.421498462" Sep 18 17:15:56.185254 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:56.185195 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wqx8x" podStartSLOduration=67.185180317 podStartE2EDuration="1m7.185180317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:15:56.184736484 +0000 UTC m=+69.437897520" watchObservedRunningTime="2026-09-18 17:15:56.185180317 +0000 UTC m=+69.438341138" Sep 18 17:15:56.259453 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:56.259431 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:56.460223 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:56.460163 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:56.661468 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:56.661445 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:56.857509 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:56.857448 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:57.021219 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:57.021188 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:57.118939 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:57.118873 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:57.876452 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:57.876420 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:57.949377 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:57.949342 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:58.265502 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:58.265473 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:58.460171 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:58.460144 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:58.664014 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:58.663987 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:58.868804 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:58.868773 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:15:59.858612 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:15:59.858496 2560 scope.go:117] "RemoveContainer" containerID="5a29ffa870479a1326b123cf2f84e4700acbbcead141267aa92bbb1c4f5beacd" Sep 18 17:16:00.167560 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:00.167532 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:16:00.167722 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:00.167627 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" event={"ID":"5aa5aa57-bbfa-461a-b12a-e32ad883c338","Type":"ContainerStarted","Data":"aaf81b02b369a0e3610bf03c3cf6f0a9800e2414552ab11f2577de757fdaa123"} Sep 18 17:16:00.168071 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:00.168036 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:16:00.189633 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:00.189590 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" podStartSLOduration=23.833411048 podStartE2EDuration="26.189576926s" totalImagesPullingTime="2.356165878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:00.189400465 +0000 UTC m=+73.442561282" watchObservedRunningTime="2026-09-18 17:16:00.189576926 +0000 UTC m=+73.442737725" Sep 18 17:16:00.718117 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:00.718091 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-dp86w" Sep 18 17:16:00.895901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:00.895856 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:00.992769 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:00.992656 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:01.094510 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:01.094471 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:01.245175 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:01.245094 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:01.485192 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:01.485161 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:01.587999 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:01.587928 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:02.311211 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:02.311180 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:02.420066 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:02.420031 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:02.515356 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:02.515323 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:02.615526 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:02.615448 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:02.716240 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:02.716209 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:03.809225 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:03.809192 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:03.909657 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:03.909627 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:04.246974 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.246946 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6"] Sep 18 17:16:04.248796 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.248781 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" Sep 18 17:16:04.252798 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.252775 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 18 17:16:04.252798 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.252782 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 18 17:16:04.252995 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.252834 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 18 17:16:04.252995 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.252864 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 18 17:16:04.267921 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.267893 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6"] Sep 18 17:16:04.274781 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.274760 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d6dc57758-jwp4d"] Sep 18 17:16:04.274958 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:16:04.274942 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" podUID="acaef214-5866-40a6-bb5e-dce035b34f5b" Sep 18 17:16:04.306108 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.306078 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-d2c2h"] Sep 18 17:16:04.307935 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.307921 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-d2c2h" Sep 18 17:16:04.310560 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.310536 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 18 17:16:04.310668 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.310539 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7t6ps\"" Sep 18 17:16:04.310842 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.310826 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 18 17:16:04.325535 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.325510 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-d2c2h"] Sep 18 17:16:04.358693 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.358651 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af66573b-c8fa-4ee7-aaa1-bb277f707599-tmp\") pod \"klusterlet-addon-workmgr-978dc5d48-j9hq6\" (UID: \"af66573b-c8fa-4ee7-aaa1-bb277f707599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" Sep 18 17:16:04.358862 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.358700 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8s2d\" (UniqueName: \"kubernetes.io/projected/af66573b-c8fa-4ee7-aaa1-bb277f707599-kube-api-access-n8s2d\") pod \"klusterlet-addon-workmgr-978dc5d48-j9hq6\" (UID: \"af66573b-c8fa-4ee7-aaa1-bb277f707599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" Sep 18 17:16:04.358862 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.358752 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4g67\" (UniqueName: \"kubernetes.io/projected/693f2b8e-0aad-43cd-8e2c-d5f9cd57b949-kube-api-access-w4g67\") pod \"downloads-6594f4c75-d2c2h\" (UID: \"693f2b8e-0aad-43cd-8e2c-d5f9cd57b949\") " pod="openshift-console/downloads-6594f4c75-d2c2h" Sep 18 17:16:04.358862 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.358790 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/af66573b-c8fa-4ee7-aaa1-bb277f707599-klusterlet-config\") pod \"klusterlet-addon-workmgr-978dc5d48-j9hq6\" (UID: \"af66573b-c8fa-4ee7-aaa1-bb277f707599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" Sep 18 17:16:04.411621 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.411589 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:04.459153 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.459123 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4g67\" (UniqueName: \"kubernetes.io/projected/693f2b8e-0aad-43cd-8e2c-d5f9cd57b949-kube-api-access-w4g67\") pod \"downloads-6594f4c75-d2c2h\" (UID: \"693f2b8e-0aad-43cd-8e2c-d5f9cd57b949\") " pod="openshift-console/downloads-6594f4c75-d2c2h" Sep 18 17:16:04.459286 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.459163 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/af66573b-c8fa-4ee7-aaa1-bb277f707599-klusterlet-config\") pod \"klusterlet-addon-workmgr-978dc5d48-j9hq6\" (UID: \"af66573b-c8fa-4ee7-aaa1-bb277f707599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" Sep 18 17:16:04.459286 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.459246 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af66573b-c8fa-4ee7-aaa1-bb277f707599-tmp\") pod \"klusterlet-addon-workmgr-978dc5d48-j9hq6\" (UID: \"af66573b-c8fa-4ee7-aaa1-bb277f707599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" Sep 18 17:16:04.459286 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.459269 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8s2d\" (UniqueName: \"kubernetes.io/projected/af66573b-c8fa-4ee7-aaa1-bb277f707599-kube-api-access-n8s2d\") pod \"klusterlet-addon-workmgr-978dc5d48-j9hq6\" (UID: \"af66573b-c8fa-4ee7-aaa1-bb277f707599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" Sep 18 17:16:04.459607 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.459587 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af66573b-c8fa-4ee7-aaa1-bb277f707599-tmp\") pod \"klusterlet-addon-workmgr-978dc5d48-j9hq6\" (UID: \"af66573b-c8fa-4ee7-aaa1-bb277f707599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" Sep 18 17:16:04.461684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.461663 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/af66573b-c8fa-4ee7-aaa1-bb277f707599-klusterlet-config\") pod \"klusterlet-addon-workmgr-978dc5d48-j9hq6\" (UID: \"af66573b-c8fa-4ee7-aaa1-bb277f707599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" Sep 18 17:16:04.468025 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.468000 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8s2d\" (UniqueName: \"kubernetes.io/projected/af66573b-c8fa-4ee7-aaa1-bb277f707599-kube-api-access-n8s2d\") pod \"klusterlet-addon-workmgr-978dc5d48-j9hq6\" (UID: \"af66573b-c8fa-4ee7-aaa1-bb277f707599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" Sep 18 17:16:04.472449 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.472427 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4g67\" (UniqueName: \"kubernetes.io/projected/693f2b8e-0aad-43cd-8e2c-d5f9cd57b949-kube-api-access-w4g67\") pod \"downloads-6594f4c75-d2c2h\" (UID: \"693f2b8e-0aad-43cd-8e2c-d5f9cd57b949\") " pod="openshift-console/downloads-6594f4c75-d2c2h" Sep 18 17:16:04.499470 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.499411 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:04.557639 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.557606 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" Sep 18 17:16:04.617430 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.617405 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-d2c2h" Sep 18 17:16:04.825813 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.825620 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6"] Sep 18 17:16:04.828171 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:04.828138 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf66573b_c8fa_4ee7_aaa1_bb277f707599.slice/crio-161d4a6f2622aa0889cb69f37779f3dd6829b08dbda38b975ab4acb0d94d085d WatchSource:0}: Error finding container 161d4a6f2622aa0889cb69f37779f3dd6829b08dbda38b975ab4acb0d94d085d: Status 404 returned error can't find the container with id 161d4a6f2622aa0889cb69f37779f3dd6829b08dbda38b975ab4acb0d94d085d Sep 18 17:16:04.897997 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:04.897931 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-d2c2h"] Sep 18 17:16:04.899972 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:04.899942 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693f2b8e_0aad_43cd_8e2c_d5f9cd57b949.slice/crio-b028979c2296ee3364c9887a50344b28bbd5a75982c9bfdf0b08faa21b60028a WatchSource:0}: Error finding container b028979c2296ee3364c9887a50344b28bbd5a75982c9bfdf0b08faa21b60028a: Status 404 returned error can't find the container with id b028979c2296ee3364c9887a50344b28bbd5a75982c9bfdf0b08faa21b60028a Sep 18 17:16:05.183942 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.183911 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" event={"ID":"af66573b-c8fa-4ee7-aaa1-bb277f707599","Type":"ContainerStarted","Data":"161d4a6f2622aa0889cb69f37779f3dd6829b08dbda38b975ab4acb0d94d085d"} Sep 18 17:16:05.184968 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.184946 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-d2c2h" event={"ID":"693f2b8e-0aad-43cd-8e2c-d5f9cd57b949","Type":"ContainerStarted","Data":"b028979c2296ee3364c9887a50344b28bbd5a75982c9bfdf0b08faa21b60028a"} Sep 18 17:16:05.185069 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.184985 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:16:05.192108 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.192087 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:16:05.265360 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.265323 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-certificates\") pod \"acaef214-5866-40a6-bb5e-dce035b34f5b\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " Sep 18 17:16:05.265554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.265375 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acaef214-5866-40a6-bb5e-dce035b34f5b-installation-pull-secrets\") pod \"acaef214-5866-40a6-bb5e-dce035b34f5b\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " Sep 18 17:16:05.265554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.265410 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acaef214-5866-40a6-bb5e-dce035b34f5b-trusted-ca\") pod \"acaef214-5866-40a6-bb5e-dce035b34f5b\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " Sep 18 17:16:05.265554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.265445 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acaef214-5866-40a6-bb5e-dce035b34f5b-ca-trust-extracted\") pod \"acaef214-5866-40a6-bb5e-dce035b34f5b\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " Sep 18 17:16:05.265554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.265463 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-bound-sa-token\") pod \"acaef214-5866-40a6-bb5e-dce035b34f5b\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " Sep 18 17:16:05.265554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.265489 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2dsm\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-kube-api-access-p2dsm\") pod \"acaef214-5866-40a6-bb5e-dce035b34f5b\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " Sep 18 17:16:05.265554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.265528 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acaef214-5866-40a6-bb5e-dce035b34f5b-image-registry-private-configuration\") pod \"acaef214-5866-40a6-bb5e-dce035b34f5b\" (UID: \"acaef214-5866-40a6-bb5e-dce035b34f5b\") " Sep 18 17:16:05.265856 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.265761 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "acaef214-5866-40a6-bb5e-dce035b34f5b" (UID: "acaef214-5866-40a6-bb5e-dce035b34f5b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:16:05.265856 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.265788 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acaef214-5866-40a6-bb5e-dce035b34f5b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "acaef214-5866-40a6-bb5e-dce035b34f5b" (UID: "acaef214-5866-40a6-bb5e-dce035b34f5b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:16:05.266091 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.266068 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acaef214-5866-40a6-bb5e-dce035b34f5b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "acaef214-5866-40a6-bb5e-dce035b34f5b" (UID: "acaef214-5866-40a6-bb5e-dce035b34f5b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:16:05.267732 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.267704 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acaef214-5866-40a6-bb5e-dce035b34f5b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "acaef214-5866-40a6-bb5e-dce035b34f5b" (UID: "acaef214-5866-40a6-bb5e-dce035b34f5b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:16:05.267818 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.267785 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-kube-api-access-p2dsm" (OuterVolumeSpecName: "kube-api-access-p2dsm") pod "acaef214-5866-40a6-bb5e-dce035b34f5b" (UID: "acaef214-5866-40a6-bb5e-dce035b34f5b"). InnerVolumeSpecName "kube-api-access-p2dsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:16:05.268024 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.268001 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acaef214-5866-40a6-bb5e-dce035b34f5b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "acaef214-5866-40a6-bb5e-dce035b34f5b" (UID: "acaef214-5866-40a6-bb5e-dce035b34f5b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:16:05.268067 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.268006 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "acaef214-5866-40a6-bb5e-dce035b34f5b" (UID: "acaef214-5866-40a6-bb5e-dce035b34f5b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:16:05.366729 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.366696 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acaef214-5866-40a6-bb5e-dce035b34f5b-trusted-ca\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:05.366729 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.366721 2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acaef214-5866-40a6-bb5e-dce035b34f5b-ca-trust-extracted\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:05.366729 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.366731 2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-bound-sa-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:05.366952 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.366740 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2dsm\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-kube-api-access-p2dsm\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:05.366952 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.366749 2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acaef214-5866-40a6-bb5e-dce035b34f5b-image-registry-private-configuration\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:05.366952 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.366760 2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-certificates\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:05.366952 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:05.366768 2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acaef214-5866-40a6-bb5e-dce035b34f5b-installation-pull-secrets\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:06.154232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:06.154195 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qqglr" Sep 18 17:16:06.190319 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:06.189492 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d6dc57758-jwp4d" Sep 18 17:16:06.317348 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:06.317271 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d6dc57758-jwp4d"] Sep 18 17:16:06.334123 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:06.334055 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d6dc57758-jwp4d"] Sep 18 17:16:06.375772 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:06.375163 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:16:06.384126 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:06.384031 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a4216580-a12b-49e1-b2e1-607171a27a96-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xbwf\" (UID: \"a4216580-a12b-49e1-b2e1-607171a27a96\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:16:06.476320 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:06.476211 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acaef214-5866-40a6-bb5e-dce035b34f5b-registry-tls\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:06.611671 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:06.611641 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8zkxl\"" Sep 18 17:16:06.619574 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:06.619346 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" Sep 18 17:16:06.864652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:06.864561 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acaef214-5866-40a6-bb5e-dce035b34f5b" path="/var/lib/kubelet/pods/acaef214-5866-40a6-bb5e-dce035b34f5b/volumes" Sep 18 17:16:07.355052 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:07.355021 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:07.418235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:07.418156 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:07.585737 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:07.585377 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf"] Sep 18 17:16:07.588773 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:07.588738 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4216580_a12b_49e1_b2e1_607171a27a96.slice/crio-c5dc4222a1ec70139ee86aa06a3078fcda8c49c8f8deac56ec1fbc08e3640186 WatchSource:0}: Error finding container c5dc4222a1ec70139ee86aa06a3078fcda8c49c8f8deac56ec1fbc08e3640186: Status 404 returned error can't find the container with id c5dc4222a1ec70139ee86aa06a3078fcda8c49c8f8deac56ec1fbc08e3640186 Sep 18 17:16:07.621849 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:07.621719 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:07.686267 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:07.686237 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:08.208662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:08.208558 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" event={"ID":"a4216580-a12b-49e1-b2e1-607171a27a96","Type":"ContainerStarted","Data":"c5dc4222a1ec70139ee86aa06a3078fcda8c49c8f8deac56ec1fbc08e3640186"} Sep 18 17:16:09.214886 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:09.214845 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" event={"ID":"af66573b-c8fa-4ee7-aaa1-bb277f707599","Type":"ContainerStarted","Data":"460a3ec47cf1d47290b97e4a0d15b07424215281cf9e66eaf7de697b4cdb1d88"} Sep 18 17:16:09.215339 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:09.215114 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" Sep 18 17:16:09.216879 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:09.216855 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" Sep 18 17:16:09.232418 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:09.232365 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-978dc5d48-j9hq6" podStartSLOduration=1.039054968 podStartE2EDuration="5.232349052s" totalImagesPullingTime="4.193294084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:09.231260695 +0000 UTC m=+82.484421537" watchObservedRunningTime="2026-09-18 17:16:09.232349052 +0000 UTC m=+82.485509868" Sep 18 17:16:09.809802 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:09.809769 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:10.071139 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.071055 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ff47cf8c4-czrgs"] Sep 18 17:16:10.075429 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.074497 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.077703 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.077683 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 18 17:16:10.078021 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.078000 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 18 17:16:10.079372 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.078853 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 18 17:16:10.079372 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.079000 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kp6ft\"" Sep 18 17:16:10.079372 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.079019 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 18 17:16:10.079372 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.079002 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 18 17:16:10.085159 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.085136 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ff47cf8c4-czrgs"] Sep 18 17:16:10.208347 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.208315 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-config\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.208347 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.208351 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-oauth-serving-cert\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.208590 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.208386 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-serving-cert\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.208590 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.208430 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-service-ca\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.208590 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.208464 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-oauth-config\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.208590 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.208492 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9594t\" (UniqueName: \"kubernetes.io/projected/eb4ea6df-f412-483b-ad31-4dca01458ba6-kube-api-access-9594t\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.309353 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.309308 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-oauth-serving-cert\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.309821 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.309382 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-serving-cert\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.309821 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.309428 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-service-ca\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.309821 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.309456 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-oauth-config\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.309821 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.309480 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9594t\" (UniqueName: \"kubernetes.io/projected/eb4ea6df-f412-483b-ad31-4dca01458ba6-kube-api-access-9594t\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.309821 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.309604 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-config\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.310116 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.310084 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-oauth-serving-cert\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.310174 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.310126 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-service-ca\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.310281 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.310256 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-config\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.312705 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.312678 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-serving-cert\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.313142 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.313121 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-oauth-config\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.319193 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.319164 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9594t\" (UniqueName: \"kubernetes.io/projected/eb4ea6df-f412-483b-ad31-4dca01458ba6-kube-api-access-9594t\") pod \"console-7ff47cf8c4-czrgs\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.387414 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.387332 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:10.705004 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.704946 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ff47cf8c4-czrgs"] Sep 18 17:16:10.708588 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:10.708550 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4ea6df_f412_483b_ad31_4dca01458ba6.slice/crio-b11f7c365cb104742d30a419c1134cb7594e28ab4a959997cb7aca73e4e5542f WatchSource:0}: Error finding container b11f7c365cb104742d30a419c1134cb7594e28ab4a959997cb7aca73e4e5542f: Status 404 returned error can't find the container with id b11f7c365cb104742d30a419c1134cb7594e28ab4a959997cb7aca73e4e5542f Sep 18 17:16:10.801787 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.801758 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:10.903588 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.903535 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:10.915525 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.915484 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:16:10.918834 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.918777 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c6aba87-34e8-41a8-9157-96b93bd9dbda-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9kbh\" (UID: \"6c6aba87-34e8-41a8-9157-96b93bd9dbda\") " pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:16:10.976612 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.976540 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8stwn"] Sep 18 17:16:10.978824 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.978797 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8stwn" Sep 18 17:16:10.979682 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.979656 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:10.981530 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.981507 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 18 17:16:10.981872 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.981851 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ghmsh\"" Sep 18 17:16:10.988836 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:10.988777 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8stwn"] Sep 18 17:16:11.117321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:11.117267 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/29a6d4d4-6c68-4fde-b929-377ff159b020-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-8stwn\" (UID: \"29a6d4d4-6c68-4fde-b929-377ff159b020\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8stwn" Sep 18 17:16:11.161329 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:11.161281 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2sl55\"" Sep 18 17:16:11.170028 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:11.169628 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s9kbh" Sep 18 17:16:11.219470 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:11.218542 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/29a6d4d4-6c68-4fde-b929-377ff159b020-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-8stwn\" (UID: \"29a6d4d4-6c68-4fde-b929-377ff159b020\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8stwn" Sep 18 17:16:11.223925 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:11.223887 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/29a6d4d4-6c68-4fde-b929-377ff159b020-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-8stwn\" (UID: \"29a6d4d4-6c68-4fde-b929-377ff159b020\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8stwn" Sep 18 17:16:11.232195 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:11.231409 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" event={"ID":"a4216580-a12b-49e1-b2e1-607171a27a96","Type":"ContainerStarted","Data":"ed4c86368f84b7797af5668bbc01fb1845fa8a91729b52a533947fe59e4cb4f6"} Sep 18 17:16:11.234929 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:11.234862 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff47cf8c4-czrgs" event={"ID":"eb4ea6df-f412-483b-ad31-4dca01458ba6","Type":"ContainerStarted","Data":"b11f7c365cb104742d30a419c1134cb7594e28ab4a959997cb7aca73e4e5542f"} Sep 18 17:16:11.256976 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:11.256396 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xbwf" podStartSLOduration=34.388559662 podStartE2EDuration="37.256377602s" totalImagesPullingTime="2.86781794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:11.253345724 +0000 UTC m=+84.506506565" watchObservedRunningTime="2026-09-18 17:16:11.256377602 +0000 UTC m=+84.509538419" Sep 18 17:16:11.257619 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:11.257544 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:11.291855 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:11.291563 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8stwn" Sep 18 17:16:11.488141 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:11.488017 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s9kbh"] Sep 18 17:16:11.494311 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:11.494259 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6aba87_34e8_41a8_9157_96b93bd9dbda.slice/crio-4ca4f8958ea1d8ae3ce98367f2f92873733dfcec009d2fb513c106edb5ba0395 WatchSource:0}: Error finding container 4ca4f8958ea1d8ae3ce98367f2f92873733dfcec009d2fb513c106edb5ba0395: Status 404 returned error can't find the container with id 4ca4f8958ea1d8ae3ce98367f2f92873733dfcec009d2fb513c106edb5ba0395 Sep 18 17:16:11.635521 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:11.634995 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8stwn"] Sep 18 17:16:11.642381 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:11.642348 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a6d4d4_6c68_4fde_b929_377ff159b020.slice/crio-e9000af1e3ffc7dee944042edba9dc804663600091b5bad7264704446796ae40 WatchSource:0}: Error finding container e9000af1e3ffc7dee944042edba9dc804663600091b5bad7264704446796ae40: Status 404 returned error can't find the container with id e9000af1e3ffc7dee944042edba9dc804663600091b5bad7264704446796ae40 Sep 18 17:16:12.243035 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:12.242918 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s9kbh" event={"ID":"6c6aba87-34e8-41a8-9157-96b93bd9dbda","Type":"ContainerStarted","Data":"257be3da3cb339e50ca5c04137c738a08cfc54f676aabb8a8d28a11784a58199"} Sep 18 17:16:12.243035 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:12.242987 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s9kbh" event={"ID":"6c6aba87-34e8-41a8-9157-96b93bd9dbda","Type":"ContainerStarted","Data":"4ca4f8958ea1d8ae3ce98367f2f92873733dfcec009d2fb513c106edb5ba0395"} Sep 18 17:16:12.246051 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:12.245985 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8stwn" event={"ID":"29a6d4d4-6c68-4fde-b929-377ff159b020","Type":"ContainerStarted","Data":"e9000af1e3ffc7dee944042edba9dc804663600091b5bad7264704446796ae40"} Sep 18 17:16:15.333747 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:15.333713 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:15.435190 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:15.435151 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:15.863264 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:15.863236 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:15.951231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:15.951202 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:17.609628 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:17.609589 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:17.678384 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:17.678348 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:20.217194 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:20.217157 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:20.318156 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:20.318123 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:21.220732 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:21.220697 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:21.315923 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:21.315892 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:22.820256 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:22.820217 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:22.920180 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:22.920144 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:23.288995 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:23.288960 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-d2c2h" event={"ID":"693f2b8e-0aad-43cd-8e2c-d5f9cd57b949","Type":"ContainerStarted","Data":"c204136b74ddbab4152a0103d65575138738a8851e311df8025f3d18b0982129"} Sep 18 17:16:23.289178 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:23.289027 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-d2c2h" Sep 18 17:16:23.290585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:23.290561 2560 patch_prober.go:28] interesting pod/downloads-6594f4c75-d2c2h container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused" start-of-body= Sep 18 17:16:23.290680 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:23.290604 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6594f4c75-d2c2h" podUID="693f2b8e-0aad-43cd-8e2c-d5f9cd57b949" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.15:8080/\": dial tcp 10.134.0.15:8080: connect: connection refused" Sep 18 17:16:23.291072 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:23.291047 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s9kbh" event={"ID":"6c6aba87-34e8-41a8-9157-96b93bd9dbda","Type":"ContainerStarted","Data":"cc51f4d90f79a1d335a703bb00caaa5429addaa21d1501415eb7f91c30f42359"} Sep 18 17:16:23.292673 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:23.292652 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8stwn" event={"ID":"29a6d4d4-6c68-4fde-b929-377ff159b020","Type":"ContainerStarted","Data":"58ef58deca8d5eb6cffffa6b525a5575026eb6838e76eedf794ccada6e61c10a"} Sep 18 17:16:23.292860 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:23.292840 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8stwn" Sep 18 17:16:23.298655 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:23.298633 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8stwn" Sep 18 17:16:23.327842 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:23.327732 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-d2c2h" podStartSLOduration=1.059841501 podStartE2EDuration="19.327703954s" totalImagesPullingTime="18.267862453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:23.31096134 +0000 UTC m=+96.564122157" watchObservedRunningTime="2026-09-18 17:16:23.327703954 +0000 UTC m=+96.580864773" Sep 18 17:16:23.328431 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:23.328391 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-8stwn" podStartSLOduration=9.262840212 podStartE2EDuration="13.328379375s" totalImagesPullingTime="4.065539163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:23.327814617 +0000 UTC m=+96.580975434" watchObservedRunningTime="2026-09-18 17:16:23.328379375 +0000 UTC m=+96.581540193" Sep 18 17:16:24.004996 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.004954 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:24.040022 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.039983 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt"] Sep 18 17:16:24.042539 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.042514 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.046968 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.046929 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 18 17:16:24.047400 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.047283 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 18 17:16:24.047850 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.047499 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cxvzm\"" Sep 18 17:16:24.047850 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.047688 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 18 17:16:24.053168 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.053052 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt"] Sep 18 17:16:24.073448 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.073415 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:24.139824 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.139784 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8692e40d-87b0-48dc-ae8d-0f9dc1592331-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-nt5qt\" (UID: \"8692e40d-87b0-48dc-ae8d-0f9dc1592331\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.139993 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.139845 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8692e40d-87b0-48dc-ae8d-0f9dc1592331-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-nt5qt\" (UID: \"8692e40d-87b0-48dc-ae8d-0f9dc1592331\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.139993 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.139947 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrvzl\" (UniqueName: \"kubernetes.io/projected/8692e40d-87b0-48dc-ae8d-0f9dc1592331-kube-api-access-hrvzl\") pod \"prometheus-operator-7bd5d56d6-nt5qt\" (UID: \"8692e40d-87b0-48dc-ae8d-0f9dc1592331\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.139993 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.139985 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8692e40d-87b0-48dc-ae8d-0f9dc1592331-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-nt5qt\" (UID: \"8692e40d-87b0-48dc-ae8d-0f9dc1592331\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.241021 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.240977 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8692e40d-87b0-48dc-ae8d-0f9dc1592331-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-nt5qt\" (UID: \"8692e40d-87b0-48dc-ae8d-0f9dc1592331\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.241211 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.241066 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8692e40d-87b0-48dc-ae8d-0f9dc1592331-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-nt5qt\" (UID: \"8692e40d-87b0-48dc-ae8d-0f9dc1592331\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.241211 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.241108 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8692e40d-87b0-48dc-ae8d-0f9dc1592331-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-nt5qt\" (UID: \"8692e40d-87b0-48dc-ae8d-0f9dc1592331\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.241211 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.241188 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrvzl\" (UniqueName: \"kubernetes.io/projected/8692e40d-87b0-48dc-ae8d-0f9dc1592331-kube-api-access-hrvzl\") pod \"prometheus-operator-7bd5d56d6-nt5qt\" (UID: \"8692e40d-87b0-48dc-ae8d-0f9dc1592331\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.241644 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:16:24.241603 2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 18 17:16:24.241742 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:16:24.241676 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8692e40d-87b0-48dc-ae8d-0f9dc1592331-prometheus-operator-tls podName:8692e40d-87b0-48dc-ae8d-0f9dc1592331 nodeName:}" failed. No retries permitted until 2026-09-18 17:16:24.741655486 +0000 UTC m=+97.994816282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/8692e40d-87b0-48dc-ae8d-0f9dc1592331-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-nt5qt" (UID: "8692e40d-87b0-48dc-ae8d-0f9dc1592331") : secret "prometheus-operator-tls" not found Sep 18 17:16:24.242366 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.242317 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8692e40d-87b0-48dc-ae8d-0f9dc1592331-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-nt5qt\" (UID: \"8692e40d-87b0-48dc-ae8d-0f9dc1592331\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.244847 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.244797 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8692e40d-87b0-48dc-ae8d-0f9dc1592331-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-nt5qt\" (UID: \"8692e40d-87b0-48dc-ae8d-0f9dc1592331\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.252360 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.252231 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrvzl\" (UniqueName: \"kubernetes.io/projected/8692e40d-87b0-48dc-ae8d-0f9dc1592331-kube-api-access-hrvzl\") pod \"prometheus-operator-7bd5d56d6-nt5qt\" (UID: \"8692e40d-87b0-48dc-ae8d-0f9dc1592331\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.302563 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.302010 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff47cf8c4-czrgs" event={"ID":"eb4ea6df-f412-483b-ad31-4dca01458ba6","Type":"ContainerStarted","Data":"96d34a8adb4ed312d57c9e0ded3f5d9dfc1fcd33a47a01a84cbc89360d91f0ed"} Sep 18 17:16:24.310948 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.310914 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-d2c2h" Sep 18 17:16:24.321909 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.321600 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7ff47cf8c4-czrgs" podStartSLOduration=1.7955773050000001 podStartE2EDuration="14.321579733s" totalImagesPullingTime="12.526002428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:24.320806882 +0000 UTC m=+97.573967724" watchObservedRunningTime="2026-09-18 17:16:24.321579733 +0000 UTC m=+97.574740553" Sep 18 17:16:24.415180 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.415067 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:24.746766 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.746733 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8692e40d-87b0-48dc-ae8d-0f9dc1592331-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-nt5qt\" (UID: \"8692e40d-87b0-48dc-ae8d-0f9dc1592331\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.749680 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.749644 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8692e40d-87b0-48dc-ae8d-0f9dc1592331-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-nt5qt\" (UID: \"8692e40d-87b0-48dc-ae8d-0f9dc1592331\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:24.936201 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.936167 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:24.958035 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:24.957997 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" Sep 18 17:16:25.001810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:25.001723 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:25.307421 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:25.307339 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s9kbh" event={"ID":"6c6aba87-34e8-41a8-9157-96b93bd9dbda","Type":"ContainerStarted","Data":"58f0a747e434e9f972a7465ba26ca8cf4890edbaee483e32aa95975c9a5b6c48"} Sep 18 17:16:25.341922 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:25.341871 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s9kbh" podStartSLOduration=41.343373545 podStartE2EDuration="47.341854062s" totalImagesPullingTime="5.998480517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:15:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:25.340204843 +0000 UTC m=+98.593365679" watchObservedRunningTime="2026-09-18 17:16:25.341854062 +0000 UTC m=+98.595014923" Sep 18 17:16:25.345892 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:25.345858 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:25.361367 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:25.360871 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt"] Sep 18 17:16:25.364249 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:25.364214 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8692e40d_87b0_48dc_ae8d_0f9dc1592331.slice/crio-26d69d32f07804c2f49059f819a08a0c2b348a87faccb3ff0df650f36b442132 WatchSource:0}: Error finding container 26d69d32f07804c2f49059f819a08a0c2b348a87faccb3ff0df650f36b442132: Status 404 returned error can't find the container with id 26d69d32f07804c2f49059f819a08a0c2b348a87faccb3ff0df650f36b442132 Sep 18 17:16:25.627391 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:25.627314 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:25.730961 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:25.730915 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:26.300202 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:26.300169 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:26.312721 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:26.312686 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" event={"ID":"8692e40d-87b0-48dc-ae8d-0f9dc1592331","Type":"ContainerStarted","Data":"26d69d32f07804c2f49059f819a08a0c2b348a87faccb3ff0df650f36b442132"} Sep 18 17:16:26.451093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:26.451051 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:26.524758 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:26.524724 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:27.012782 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:27.012743 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:27.113760 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:27.113728 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:27.153594 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:27.153563 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wqx8x" Sep 18 17:16:27.720851 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:27.720812 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:28.323228 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:28.323178 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" event={"ID":"8692e40d-87b0-48dc-ae8d-0f9dc1592331","Type":"ContainerStarted","Data":"eabd8d475ba046715725eb1ef7b85721ca0efd97b763e51ecdca3f5c7cbb2376"} Sep 18 17:16:28.323418 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:28.323246 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" event={"ID":"8692e40d-87b0-48dc-ae8d-0f9dc1592331","Type":"ContainerStarted","Data":"9307cfeff9b00ada79aff7b225e195e5c89a02f6000b08a7b5df19f29bf4fc0f"} Sep 18 17:16:28.341234 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:28.341183 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nt5qt" podStartSLOduration=2.337970435 podStartE2EDuration="4.341165206s" totalImagesPullingTime="2.003194771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:28.340626539 +0000 UTC m=+101.593787357" watchObservedRunningTime="2026-09-18 17:16:28.341165206 +0000 UTC m=+101.594326024" Sep 18 17:16:29.302256 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:29.302217 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ff47cf8c4-czrgs"] Sep 18 17:16:30.151832 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.151796 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:30.358180 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.358138 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:30.388474 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.388441 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:30.439804 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.439727 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c"] Sep 18 17:16:30.445854 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.445825 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:30.465152 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.464409 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.465450 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.465431 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c"] Sep 18 17:16:30.465614 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.465599 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-d2mw7"] Sep 18 17:16:30.468146 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.468120 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 18 17:16:30.468563 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.468537 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xz6hq\"" Sep 18 17:16:30.468824 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.468805 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 18 17:16:30.489967 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.489932 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gdwtp"] Sep 18 17:16:30.490202 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.490143 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.495492 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.495464 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 18 17:16:30.495644 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.495498 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qwzdr\"" Sep 18 17:16:30.495644 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.495580 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 18 17:16:30.495644 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.495464 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 18 17:16:30.497062 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.497034 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2df05aae-dbe0-4100-a2ba-da02010f7dd9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-qmv9c\" (UID: \"2df05aae-dbe0-4100-a2ba-da02010f7dd9\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.497178 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.497120 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2df05aae-dbe0-4100-a2ba-da02010f7dd9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-qmv9c\" (UID: \"2df05aae-dbe0-4100-a2ba-da02010f7dd9\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.497178 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.497165 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2df05aae-dbe0-4100-a2ba-da02010f7dd9-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-qmv9c\" (UID: \"2df05aae-dbe0-4100-a2ba-da02010f7dd9\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.497281 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.497224 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj9tg\" (UniqueName: \"kubernetes.io/projected/2df05aae-dbe0-4100-a2ba-da02010f7dd9-kube-api-access-cj9tg\") pod \"openshift-state-metrics-86c5bc79d6-qmv9c\" (UID: \"2df05aae-dbe0-4100-a2ba-da02010f7dd9\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.508510 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.508488 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.509087 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.508362 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-d2mw7"] Sep 18 17:16:30.511241 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.511219 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 18 17:16:30.511605 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.511582 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 18 17:16:30.511799 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.511776 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jmw5s\"" Sep 18 17:16:30.511893 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.511787 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 18 17:16:30.551768 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.551732 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:30.597941 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.597901 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2df05aae-dbe0-4100-a2ba-da02010f7dd9-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-qmv9c\" (UID: \"2df05aae-dbe0-4100-a2ba-da02010f7dd9\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.598122 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.597955 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn976\" (UniqueName: \"kubernetes.io/projected/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-kube-api-access-xn976\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.598122 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.597976 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-textfile\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.598122 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598091 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-accelerators-collector-config\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.598324 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598130 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtgsg\" (UniqueName: \"kubernetes.io/projected/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-kube-api-access-gtgsg\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.598324 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598194 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.598324 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598252 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj9tg\" (UniqueName: \"kubernetes.io/projected/2df05aae-dbe0-4100-a2ba-da02010f7dd9-kube-api-access-cj9tg\") pod \"openshift-state-metrics-86c5bc79d6-qmv9c\" (UID: \"2df05aae-dbe0-4100-a2ba-da02010f7dd9\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.598324 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598282 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-tls\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.598585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598331 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.598585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598366 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-metrics-client-ca\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.598585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598390 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.598585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598436 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-root\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.598585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598477 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.598585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598508 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.598585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598537 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.598892 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598595 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2df05aae-dbe0-4100-a2ba-da02010f7dd9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-qmv9c\" (UID: \"2df05aae-dbe0-4100-a2ba-da02010f7dd9\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.598892 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598693 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2df05aae-dbe0-4100-a2ba-da02010f7dd9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-qmv9c\" (UID: \"2df05aae-dbe0-4100-a2ba-da02010f7dd9\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.598892 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598726 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-sys\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.598892 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598752 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-wtmp\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.598892 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.598757 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2df05aae-dbe0-4100-a2ba-da02010f7dd9-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-qmv9c\" (UID: \"2df05aae-dbe0-4100-a2ba-da02010f7dd9\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.601572 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.601525 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2df05aae-dbe0-4100-a2ba-da02010f7dd9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-qmv9c\" (UID: \"2df05aae-dbe0-4100-a2ba-da02010f7dd9\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.601572 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.601561 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2df05aae-dbe0-4100-a2ba-da02010f7dd9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-qmv9c\" (UID: \"2df05aae-dbe0-4100-a2ba-da02010f7dd9\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.606618 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.606597 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj9tg\" (UniqueName: \"kubernetes.io/projected/2df05aae-dbe0-4100-a2ba-da02010f7dd9-kube-api-access-cj9tg\") pod \"openshift-state-metrics-86c5bc79d6-qmv9c\" (UID: \"2df05aae-dbe0-4100-a2ba-da02010f7dd9\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.646227 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.646192 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:30.700179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700097 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-metrics-client-ca\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.700179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700139 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.700179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700162 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-root\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.700495 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700193 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.700495 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700242 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.700495 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700254 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-root\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.700495 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700267 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.700495 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700398 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-sys\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.700495 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700428 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-wtmp\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.700495 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700486 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn976\" (UniqueName: \"kubernetes.io/projected/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-kube-api-access-xn976\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.700848 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700514 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-textfile\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.700848 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700540 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-accelerators-collector-config\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.700848 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700569 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgsg\" (UniqueName: \"kubernetes.io/projected/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-kube-api-access-gtgsg\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.700848 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700593 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.700848 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700632 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-tls\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.700848 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700661 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.700848 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700770 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-sys\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.700848 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700799 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-wtmp\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.701227 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700979 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-textfile\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.701227 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.700992 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.701353 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.701316 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-accelerators-collector-config\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.701502 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.701463 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.701658 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.701626 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.702061 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.702037 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-metrics-client-ca\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.703748 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.703729 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-tls\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.704026 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.704001 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.704126 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.704084 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.704407 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.704387 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.711941 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.711914 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtgsg\" (UniqueName: \"kubernetes.io/projected/183d465b-7662-46b0-b2d7-9c5a6fc9f82a-kube-api-access-gtgsg\") pod \"node-exporter-gdwtp\" (UID: \"183d465b-7662-46b0-b2d7-9c5a6fc9f82a\") " pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.712190 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.712169 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn976\" (UniqueName: \"kubernetes.io/projected/09d4fbe4-81ee-480e-a266-c1f65e06b8ff-kube-api-access-xn976\") pod \"kube-state-metrics-84c885658c-d2mw7\" (UID: \"09d4fbe4-81ee-480e-a266-c1f65e06b8ff\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.776957 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.776920 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" Sep 18 17:16:30.802374 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.802336 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" Sep 18 17:16:30.820086 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.820055 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gdwtp" Sep 18 17:16:30.821235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.821187 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:30.965122 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:30.964979 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:31.049999 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.049962 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:31.111067 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.111014 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c"] Sep 18 17:16:31.113867 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:31.113838 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df05aae_dbe0_4100_a2ba_da02010f7dd9.slice/crio-53d992c9f8f9ce99484e8ee13356fd7562a712cf578393649be701673632ebe3 WatchSource:0}: Error finding container 53d992c9f8f9ce99484e8ee13356fd7562a712cf578393649be701673632ebe3: Status 404 returned error can't find the container with id 53d992c9f8f9ce99484e8ee13356fd7562a712cf578393649be701673632ebe3 Sep 18 17:16:31.125209 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.125180 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:31.151918 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.151829 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-d2mw7"] Sep 18 17:16:31.156272 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:31.156232 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d4fbe4_81ee_480e_a266_c1f65e06b8ff.slice/crio-8da3e599e458a19c99e437b3d1b1e2fc7a59cacf2362664b7a7536babba174fd WatchSource:0}: Error finding container 8da3e599e458a19c99e437b3d1b1e2fc7a59cacf2362664b7a7536babba174fd: Status 404 returned error can't find the container with id 8da3e599e458a19c99e437b3d1b1e2fc7a59cacf2362664b7a7536babba174fd Sep 18 17:16:31.336974 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.336938 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" event={"ID":"09d4fbe4-81ee-480e-a266-c1f65e06b8ff","Type":"ContainerStarted","Data":"8da3e599e458a19c99e437b3d1b1e2fc7a59cacf2362664b7a7536babba174fd"} Sep 18 17:16:31.338878 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.338829 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" event={"ID":"2df05aae-dbe0-4100-a2ba-da02010f7dd9","Type":"ContainerStarted","Data":"dd8787bc636720c736d0d2ae17ee1c209bb3cab07ff958f6e96e4b9131feadbb"} Sep 18 17:16:31.339018 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.338914 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" event={"ID":"2df05aae-dbe0-4100-a2ba-da02010f7dd9","Type":"ContainerStarted","Data":"53d992c9f8f9ce99484e8ee13356fd7562a712cf578393649be701673632ebe3"} Sep 18 17:16:31.340426 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.340396 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gdwtp" event={"ID":"183d465b-7662-46b0-b2d7-9c5a6fc9f82a","Type":"ContainerStarted","Data":"1102f3606e712decbd0314c5cabdff9e6afb2df266357ca3aa041d6f64809af7"} Sep 18 17:16:31.418848 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.418808 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:31.510083 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.509543 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:31.510652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.510616 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 17:16:31.532482 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.532451 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.534087 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.534047 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 17:16:31.537614 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.537176 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6qphq\"" Sep 18 17:16:31.537614 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.537426 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 18 17:16:31.538582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.537597 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 18 17:16:31.538582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.537949 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 18 17:16:31.538582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.538024 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 18 17:16:31.538582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.538177 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 18 17:16:31.538582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.538258 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 18 17:16:31.538582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.538411 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 18 17:16:31.538582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.538476 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 18 17:16:31.538987 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.538645 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 18 17:16:31.612626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.611876 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.612626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.611947 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.612626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.611982 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.612626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.612007 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.612626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.612039 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.612626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.612151 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.612626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.612176 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.612626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.612202 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.612626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.612230 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-config-out\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.612626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.612373 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8wwt\" (UniqueName: \"kubernetes.io/projected/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-kube-api-access-m8wwt\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.612626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.612409 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.612626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.612437 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.612626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.612489 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-web-config\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.714899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.713690 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.714899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.713740 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.714899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.713784 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-web-config\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.714899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.713841 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.714899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.713882 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.714899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.713910 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.714899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.713932 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.714899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.713962 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.714899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.713996 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.714899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.714019 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.714899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.714050 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.714899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.714078 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-config-out\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.714899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.714099 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8wwt\" (UniqueName: \"kubernetes.io/projected/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-kube-api-access-m8wwt\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.716047 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:16:31.716019 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-alertmanager-trusted-ca-bundle podName:9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1 nodeName:}" failed. No retries permitted until 2026-09-18 17:16:32.215996295 +0000 UTC m=+105.469157091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1") : configmap references non-existent config key: ca-bundle.crt Sep 18 17:16:31.716437 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.716412 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.717677 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.717643 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.723430 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.723374 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.724028 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.723979 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.727373 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.727349 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8wwt\" (UniqueName: \"kubernetes.io/projected/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-kube-api-access-m8wwt\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.729856 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.729579 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-config-out\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.729856 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.729815 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-web-config\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.731446 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.731394 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.733244 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.733198 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.733512 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.733467 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.734026 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.733984 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.735687 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.735645 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:31.738798 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.738769 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:31.833522 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:31.833411 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:32.219664 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:32.218799 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:32.220723 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:32.220671 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1\") " pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:32.348646 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:32.348563 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" event={"ID":"2df05aae-dbe0-4100-a2ba-da02010f7dd9","Type":"ContainerStarted","Data":"8c61bc6090fada032e25ca34e300e82d1f2d16db5f61fae6d1a90f76250d881f"} Sep 18 17:16:32.450316 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:32.450271 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 18 17:16:33.453525 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.453476 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:33.503761 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.503735 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f756495df-fzz5b"] Sep 18 17:16:33.529190 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.529157 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f756495df-fzz5b"] Sep 18 17:16:33.529346 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.529304 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.532914 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.532156 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 18 17:16:33.532914 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.532156 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 18 17:16:33.532914 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.532549 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fd6kp9g03ccgk\"" Sep 18 17:16:33.533332 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.532934 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 18 17:16:33.533332 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.533180 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4cbj2\"" Sep 18 17:16:33.533443 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.533431 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 18 17:16:33.536392 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.536371 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 18 17:16:33.566452 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.566416 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:33.633617 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.633579 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-tls\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.633751 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.633651 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-grpc-tls\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.633751 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.633674 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75jj\" (UniqueName: \"kubernetes.io/projected/790f80ee-9474-4c64-842e-7a0b7cc741dc-kube-api-access-g75jj\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.633835 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.633762 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.633835 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.633805 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.633890 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.633834 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/790f80ee-9474-4c64-842e-7a0b7cc741dc-metrics-client-ca\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.633928 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.633888 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.633928 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.633918 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.734906 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.734872 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.735035 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.734921 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.735091 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.735051 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-tls\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.735140 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.735125 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-grpc-tls\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.735198 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.735164 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75jj\" (UniqueName: \"kubernetes.io/projected/790f80ee-9474-4c64-842e-7a0b7cc741dc-kube-api-access-g75jj\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.735247 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.735205 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.735327 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.735249 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.735327 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.735282 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/790f80ee-9474-4c64-842e-7a0b7cc741dc-metrics-client-ca\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.736141 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.736108 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/790f80ee-9474-4c64-842e-7a0b7cc741dc-metrics-client-ca\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.737792 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.737760 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-grpc-tls\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.737908 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.737759 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.737908 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.737879 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.738025 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.737990 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-tls\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.738181 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.738159 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.738241 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.738223 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/790f80ee-9474-4c64-842e-7a0b7cc741dc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.744580 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.744558 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75jj\" (UniqueName: \"kubernetes.io/projected/790f80ee-9474-4c64-842e-7a0b7cc741dc-kube-api-access-g75jj\") pod \"thanos-querier-6f756495df-fzz5b\" (UID: \"790f80ee-9474-4c64-842e-7a0b7cc741dc\") " pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.842686 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.842662 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:33.991127 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:33.991058 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:34.039447 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.039421 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 18 17:16:34.051894 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:34.051852 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c73bc2c_eeaa_4aa5_9473_0cadc8a7c6a1.slice/crio-10dfe9befd974efcaf5be90a700ee93bb26aa197c355e0181f1db98d0702d4ae WatchSource:0}: Error finding container 10dfe9befd974efcaf5be90a700ee93bb26aa197c355e0181f1db98d0702d4ae: Status 404 returned error can't find the container with id 10dfe9befd974efcaf5be90a700ee93bb26aa197c355e0181f1db98d0702d4ae Sep 18 17:16:34.143253 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.143222 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f756495df-fzz5b"] Sep 18 17:16:34.144170 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:34.144134 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790f80ee_9474_4c64_842e_7a0b7cc741dc.slice/crio-85bdb7575803be4a7df9365a79499cb21dba96b3cb887e448e92c143c339f779 WatchSource:0}: Error finding container 85bdb7575803be4a7df9365a79499cb21dba96b3cb887e448e92c143c339f779: Status 404 returned error can't find the container with id 85bdb7575803be4a7df9365a79499cb21dba96b3cb887e448e92c143c339f779 Sep 18 17:16:34.358656 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.358559 2560 generic.go:356] "Generic (PLEG): container finished" podID="183d465b-7662-46b0-b2d7-9c5a6fc9f82a" containerID="9dd256e066b0b6fbeb4a64ac95b4feed96abcb61c9266b9cce71ef7485fb58f4" exitCode=0 Sep 18 17:16:34.358832 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.358652 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gdwtp" event={"ID":"183d465b-7662-46b0-b2d7-9c5a6fc9f82a","Type":"ContainerDied","Data":"9dd256e066b0b6fbeb4a64ac95b4feed96abcb61c9266b9cce71ef7485fb58f4"} Sep 18 17:16:34.360150 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.360128 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1","Type":"ContainerStarted","Data":"10dfe9befd974efcaf5be90a700ee93bb26aa197c355e0181f1db98d0702d4ae"} Sep 18 17:16:34.362571 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.362546 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" event={"ID":"2df05aae-dbe0-4100-a2ba-da02010f7dd9","Type":"ContainerStarted","Data":"b0292cfe0fd9e67f63c895ae08206031b4496d9bcf33f46b18e8d6cfeb7d48d6"} Sep 18 17:16:34.363900 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.363880 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" event={"ID":"790f80ee-9474-4c64-842e-7a0b7cc741dc","Type":"ContainerStarted","Data":"85bdb7575803be4a7df9365a79499cb21dba96b3cb887e448e92c143c339f779"} Sep 18 17:16:34.366196 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.366177 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" event={"ID":"09d4fbe4-81ee-480e-a266-c1f65e06b8ff","Type":"ContainerStarted","Data":"3769b18b1abd00d602fbc1d0a7a774f90cdfe4fd6181bed6da754fe3937213a1"} Sep 18 17:16:34.366283 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.366224 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" event={"ID":"09d4fbe4-81ee-480e-a266-c1f65e06b8ff","Type":"ContainerStarted","Data":"786f4d45239b067c06f1073e82f84e7bfde0308121db2ff8be209306a617272e"} Sep 18 17:16:34.366283 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.366260 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" event={"ID":"09d4fbe4-81ee-480e-a266-c1f65e06b8ff","Type":"ContainerStarted","Data":"685f5629ed647ff4c12fde68f3a1b980bdbf6144fd8eae1765d53ab4fd42d343"} Sep 18 17:16:34.400336 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.400270 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qmv9c" podStartSLOduration=2.011831381 podStartE2EDuration="4.400255819s" totalImagesPullingTime="2.388424438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:34.399397564 +0000 UTC m=+107.652558404" watchObservedRunningTime="2026-09-18 17:16:34.400255819 +0000 UTC m=+107.653416636" Sep 18 17:16:34.421885 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.421844 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-d2mw7" podStartSLOduration=1.8168781059999999 podStartE2EDuration="4.421826678s" totalImagesPullingTime="2.604948572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:34.420014421 +0000 UTC m=+107.673175258" watchObservedRunningTime="2026-09-18 17:16:34.421826678 +0000 UTC m=+107.674987494" Sep 18 17:16:34.890709 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.890677 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5d4c6d666-2lpbp"] Sep 18 17:16:34.909270 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.909229 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:34.910496 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.910469 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d4c6d666-2lpbp"] Sep 18 17:16:34.912510 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.912321 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 18 17:16:34.914634 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.914613 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 18 17:16:34.915140 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.914972 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 18 17:16:34.915140 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.915069 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xtqcn\"" Sep 18 17:16:34.915140 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.915114 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dlpu00qtgr03n\"" Sep 18 17:16:34.915493 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.915466 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 18 17:16:34.946049 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.946008 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrm2n\" (UniqueName: \"kubernetes.io/projected/a42baf8a-7f22-47bd-9d1f-a658bc38135d-kube-api-access-vrm2n\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:34.946049 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.946052 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a42baf8a-7f22-47bd-9d1f-a658bc38135d-client-ca-bundle\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:34.946265 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.946105 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a42baf8a-7f22-47bd-9d1f-a658bc38135d-audit-log\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:34.946265 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.946183 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a42baf8a-7f22-47bd-9d1f-a658bc38135d-secret-metrics-server-client-certs\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:34.946265 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.946220 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a42baf8a-7f22-47bd-9d1f-a658bc38135d-metrics-server-audit-profiles\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:34.946265 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.946260 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a42baf8a-7f22-47bd-9d1f-a658bc38135d-secret-metrics-server-tls\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:34.946487 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:34.946287 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a42baf8a-7f22-47bd-9d1f-a658bc38135d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.046929 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.046899 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a42baf8a-7f22-47bd-9d1f-a658bc38135d-metrics-server-audit-profiles\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.047158 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.046950 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a42baf8a-7f22-47bd-9d1f-a658bc38135d-secret-metrics-server-tls\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.047158 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.046980 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a42baf8a-7f22-47bd-9d1f-a658bc38135d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.047158 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.047021 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrm2n\" (UniqueName: \"kubernetes.io/projected/a42baf8a-7f22-47bd-9d1f-a658bc38135d-kube-api-access-vrm2n\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.047158 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.047071 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a42baf8a-7f22-47bd-9d1f-a658bc38135d-client-ca-bundle\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.047158 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.047127 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a42baf8a-7f22-47bd-9d1f-a658bc38135d-audit-log\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.047465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.047191 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a42baf8a-7f22-47bd-9d1f-a658bc38135d-secret-metrics-server-client-certs\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.048067 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.047803 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a42baf8a-7f22-47bd-9d1f-a658bc38135d-audit-log\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.048067 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.048025 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a42baf8a-7f22-47bd-9d1f-a658bc38135d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.048230 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.048071 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a42baf8a-7f22-47bd-9d1f-a658bc38135d-metrics-server-audit-profiles\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.050356 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.050335 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a42baf8a-7f22-47bd-9d1f-a658bc38135d-secret-metrics-server-client-certs\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.050533 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.050512 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a42baf8a-7f22-47bd-9d1f-a658bc38135d-secret-metrics-server-tls\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.050621 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.050567 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a42baf8a-7f22-47bd-9d1f-a658bc38135d-client-ca-bundle\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.058868 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.058840 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrm2n\" (UniqueName: \"kubernetes.io/projected/a42baf8a-7f22-47bd-9d1f-a658bc38135d-kube-api-access-vrm2n\") pod \"metrics-server-5d4c6d666-2lpbp\" (UID: \"a42baf8a-7f22-47bd-9d1f-a658bc38135d\") " pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.223574 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.223532 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:35.228156 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.228100 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:35.244012 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.243258 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-w77st"] Sep 18 17:16:35.258699 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.258633 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-w77st" Sep 18 17:16:35.260021 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.259965 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-w77st"] Sep 18 17:16:35.261842 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.261800 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-d9b4c\"" Sep 18 17:16:35.262058 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.262030 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 18 17:16:35.316415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.316384 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:35.349893 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.349859 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55xmg\" (UniqueName: \"kubernetes.io/projected/6e7287c8-edd4-45d6-a36f-1b53d0f2172e-kube-api-access-55xmg\") pod \"monitoring-plugin-5984459ff8-w77st\" (UID: \"6e7287c8-edd4-45d6-a36f-1b53d0f2172e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-w77st" Sep 18 17:16:35.350217 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.349980 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e7287c8-edd4-45d6-a36f-1b53d0f2172e-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-w77st\" (UID: \"6e7287c8-edd4-45d6-a36f-1b53d0f2172e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-w77st" Sep 18 17:16:35.372609 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.372556 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gdwtp" event={"ID":"183d465b-7662-46b0-b2d7-9c5a6fc9f82a","Type":"ContainerStarted","Data":"894087656b7f2093339ec332ee1d6f2749370b6a5ae7688d599c3516fb1f4223"} Sep 18 17:16:35.372792 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.372656 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gdwtp" event={"ID":"183d465b-7662-46b0-b2d7-9c5a6fc9f82a","Type":"ContainerStarted","Data":"32dcfa94e8af995c4fe4570ce2c7bb3160e357dbaf1ad8967c7e063e35e1de96"} Sep 18 17:16:35.397071 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.397010 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gdwtp" podStartSLOduration=2.427691059 podStartE2EDuration="5.396993796s" totalImagesPullingTime="2.871087947s" totalInitContainerRuntime="98.21479ms" isStatefulPod=true podCreationTimestamp="2026-09-18 17:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:35.392828641 +0000 UTC m=+108.645989484" watchObservedRunningTime="2026-09-18 17:16:35.396993796 +0000 UTC m=+108.650154613" Sep 18 17:16:35.408310 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.408254 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:35.452172 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.451431 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55xmg\" (UniqueName: \"kubernetes.io/projected/6e7287c8-edd4-45d6-a36f-1b53d0f2172e-kube-api-access-55xmg\") pod \"monitoring-plugin-5984459ff8-w77st\" (UID: \"6e7287c8-edd4-45d6-a36f-1b53d0f2172e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-w77st" Sep 18 17:16:35.452172 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.451619 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e7287c8-edd4-45d6-a36f-1b53d0f2172e-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-w77st\" (UID: \"6e7287c8-edd4-45d6-a36f-1b53d0f2172e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-w77st" Sep 18 17:16:35.454944 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.454905 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e7287c8-edd4-45d6-a36f-1b53d0f2172e-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-w77st\" (UID: \"6e7287c8-edd4-45d6-a36f-1b53d0f2172e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-w77st" Sep 18 17:16:35.461227 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.461191 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55xmg\" (UniqueName: \"kubernetes.io/projected/6e7287c8-edd4-45d6-a36f-1b53d0f2172e-kube-api-access-55xmg\") pod \"monitoring-plugin-5984459ff8-w77st\" (UID: \"6e7287c8-edd4-45d6-a36f-1b53d0f2172e\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-w77st" Sep 18 17:16:35.498749 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.498711 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:35.574889 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.574852 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-w77st" Sep 18 17:16:35.718069 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:35.718048 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d4c6d666-2lpbp"] Sep 18 17:16:35.745409 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:35.745337 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42baf8a_7f22_47bd_9d1f_a658bc38135d.slice/crio-16f30085249d160543bf4f200464972fe47e11b786f10a1eaea70c5a8e64ca6a WatchSource:0}: Error finding container 16f30085249d160543bf4f200464972fe47e11b786f10a1eaea70c5a8e64ca6a: Status 404 returned error can't find the container with id 16f30085249d160543bf4f200464972fe47e11b786f10a1eaea70c5a8e64ca6a Sep 18 17:16:36.044868 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.044768 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-w77st"] Sep 18 17:16:36.379332 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.379235 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1","Type":"ContainerStarted","Data":"674f152483fb5bf3058123c572027dbe612ee6fbf00d5d1aac26f62943f00ccb"} Sep 18 17:16:36.380600 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.380575 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" event={"ID":"a42baf8a-7f22-47bd-9d1f-a658bc38135d","Type":"ContainerStarted","Data":"16f30085249d160543bf4f200464972fe47e11b786f10a1eaea70c5a8e64ca6a"} Sep 18 17:16:36.429057 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:36.429029 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7287c8_edd4_45d6_a36f_1b53d0f2172e.slice/crio-06abbbca0b99121eb82efa23d8cecf6158b861759231e601c1d69597b21929ae WatchSource:0}: Error finding container 06abbbca0b99121eb82efa23d8cecf6158b861759231e601c1d69597b21929ae: Status 404 returned error can't find the container with id 06abbbca0b99121eb82efa23d8cecf6158b861759231e601c1d69597b21929ae Sep 18 17:16:36.717532 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.717492 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:36.731039 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.730985 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 18 17:16:36.749972 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.749938 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.757672 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.757647 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 18 17:16:36.758043 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.758020 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 18 17:16:36.758155 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.758136 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 18 17:16:36.758212 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.758164 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 18 17:16:36.759134 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.758378 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 18 17:16:36.759134 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.758471 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 18 17:16:36.759134 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.758522 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 18 17:16:36.759134 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.758599 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 18 17:16:36.759134 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.758653 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-8s7kn\"" Sep 18 17:16:36.759134 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.758780 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 18 17:16:36.759134 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.758866 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 18 17:16:36.759134 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.758941 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 18 17:16:36.760153 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.760090 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8sq57lu66bp89\"" Sep 18 17:16:36.760528 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.760511 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 18 17:16:36.765114 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.765087 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 18 17:16:36.765961 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.765852 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 18 17:16:36.769230 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.769205 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 18 17:16:36.865444 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.865410 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.865623 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.865603 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.865756 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.865732 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.865871 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.865770 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6kxq\" (UniqueName: \"kubernetes.io/projected/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-kube-api-access-j6kxq\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.865871 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.865801 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.865871 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.865826 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.865871 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.865866 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.866075 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.865923 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.866075 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.865957 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.866075 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.866049 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.866075 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.866074 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.866273 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.866092 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.866273 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.866125 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-config\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.866273 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.866150 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.866273 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.866241 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.866521 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.866357 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.866521 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.866393 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.866521 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.866435 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.866521 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.866458 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.866521 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.866506 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967484 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967530 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967577 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967602 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967629 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967667 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-config\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967694 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967741 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967808 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967831 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967887 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967909 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967936 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967958 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.967980 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.968005 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.968924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.968038 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6kxq\" (UniqueName: \"kubernetes.io/projected/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-kube-api-access-j6kxq\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.969833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.968070 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.969833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.968094 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.969833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.968140 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.969833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.968818 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.969833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.969327 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.969833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.969431 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.972520 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.972488 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.972643 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.972535 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.973873 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.973266 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.973873 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.973364 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.973873 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.973818 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.974170 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.974146 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.980599 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.980335 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.980599 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.980560 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.981000 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.980809 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.981924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.981877 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-config\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.982020 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.981944 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6kxq\" (UniqueName: \"kubernetes.io/projected/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-kube-api-access-j6kxq\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.982450 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.982408 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.982544 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.982470 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.982671 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.982616 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.987353 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.982988 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.988162 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.988058 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:36.988162 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:36.988115 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:37.063181 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:37.062901 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:37.387361 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:37.387196 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 18 17:16:37.393339 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:37.393114 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe2fdaa_2ff6_40e2_ba63_0e383bb921cd.slice/crio-52380ac750dce32d75fc3b2836826836791a1eb7b50458767f4f4e77b6eeffa4 WatchSource:0}: Error finding container 52380ac750dce32d75fc3b2836826836791a1eb7b50458767f4f4e77b6eeffa4: Status 404 returned error can't find the container with id 52380ac750dce32d75fc3b2836826836791a1eb7b50458767f4f4e77b6eeffa4 Sep 18 17:16:37.396270 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:37.396081 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" event={"ID":"790f80ee-9474-4c64-842e-7a0b7cc741dc","Type":"ContainerStarted","Data":"3302409910dc84997c8d29b5d26e46f05ea389ce191005e27e7a153da6ebeef0"} Sep 18 17:16:37.396270 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:37.396157 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" event={"ID":"790f80ee-9474-4c64-842e-7a0b7cc741dc","Type":"ContainerStarted","Data":"a7dbf062b784699f3a4621acf5ff25279f6b0c8f621fec468efc4c516efb8ce0"} Sep 18 17:16:37.396270 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:37.396199 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" event={"ID":"790f80ee-9474-4c64-842e-7a0b7cc741dc","Type":"ContainerStarted","Data":"141a3ef65267c9dc019da31495e6c1fb5a1529726f3cb9468978aff4e19eb2a6"} Sep 18 17:16:37.400212 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:37.400175 2560 generic.go:356] "Generic (PLEG): container finished" podID="9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1" containerID="674f152483fb5bf3058123c572027dbe612ee6fbf00d5d1aac26f62943f00ccb" exitCode=0 Sep 18 17:16:37.400492 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:37.400467 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1","Type":"ContainerDied","Data":"674f152483fb5bf3058123c572027dbe612ee6fbf00d5d1aac26f62943f00ccb"} Sep 18 17:16:37.402957 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:37.402880 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-w77st" event={"ID":"6e7287c8-edd4-45d6-a36f-1b53d0f2172e","Type":"ContainerStarted","Data":"06abbbca0b99121eb82efa23d8cecf6158b861759231e601c1d69597b21929ae"} Sep 18 17:16:38.158531 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:38.158501 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:38.262358 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:38.262213 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:38.410776 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:38.410734 2560 generic.go:356] "Generic (PLEG): container finished" podID="4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd" containerID="bc2a4cd9c803672d3615c80f082c0723ada33e833b0c0e2850a683bba21bc938" exitCode=0 Sep 18 17:16:38.410929 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:38.410815 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd","Type":"ContainerDied","Data":"bc2a4cd9c803672d3615c80f082c0723ada33e833b0c0e2850a683bba21bc938"} Sep 18 17:16:38.410929 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:38.410877 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd","Type":"ContainerStarted","Data":"52380ac750dce32d75fc3b2836826836791a1eb7b50458767f4f4e77b6eeffa4"} Sep 18 17:16:39.417113 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:39.417064 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-w77st" event={"ID":"6e7287c8-edd4-45d6-a36f-1b53d0f2172e","Type":"ContainerStarted","Data":"d353954ff7df77c95534deb961fd2b459ad078400036a1dcafc686ace3fa1a6a"} Sep 18 17:16:39.417816 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:39.417797 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-w77st" Sep 18 17:16:39.420922 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:39.420886 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" event={"ID":"a42baf8a-7f22-47bd-9d1f-a658bc38135d","Type":"ContainerStarted","Data":"52c233a4c8fc4ef8e652eb2900e0cc7a5972c1a2b0aac7f4453cf002c2209708"} Sep 18 17:16:39.425982 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:39.425955 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-w77st" Sep 18 17:16:39.435769 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:39.435723 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-w77st" podStartSLOduration=1.840210948 podStartE2EDuration="4.435708943s" totalImagesPullingTime="2.595497995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:39.434568993 +0000 UTC m=+112.687729832" watchObservedRunningTime="2026-09-18 17:16:39.435708943 +0000 UTC m=+112.688869750" Sep 18 17:16:39.481929 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:39.481814 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" podStartSLOduration=2.223236515 podStartE2EDuration="5.481790181s" totalImagesPullingTime="3.258553666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:39.479833377 +0000 UTC m=+112.732994213" watchObservedRunningTime="2026-09-18 17:16:39.481790181 +0000 UTC m=+112.734950998" Sep 18 17:16:40.248815 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:40.248732 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:40.343918 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:40.343890 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:40.428926 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:40.428833 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1","Type":"ContainerStarted","Data":"8887dcb95e5203f4b5a8f6f4d6d22608d395668dfa1f9686faf760322b000348"} Sep 18 17:16:40.432709 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:40.432530 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:40.434252 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:40.434220 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" event={"ID":"790f80ee-9474-4c64-842e-7a0b7cc741dc","Type":"ContainerStarted","Data":"914beb7539c5d9ecc8143f050947fb1285290b42fd7f22405d8e0713badabc67"} Sep 18 17:16:40.434384 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:40.434315 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" event={"ID":"790f80ee-9474-4c64-842e-7a0b7cc741dc","Type":"ContainerStarted","Data":"8f32152bbfba86e119abb120e0d000e9feeb21717adefdf03724a06981783445"} Sep 18 17:16:40.434384 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:40.434370 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" event={"ID":"790f80ee-9474-4c64-842e-7a0b7cc741dc","Type":"ContainerStarted","Data":"631dc31125013ba24518b8b7054780d3bca160165655222b52bdff74435b057d"} Sep 18 17:16:41.443091 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:41.443031 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1","Type":"ContainerStarted","Data":"feb1b8f86f6bdcdf1751be2fb7fec1a437de868faa48411dbc0a5c0852e4bd42"} Sep 18 17:16:41.443642 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:41.443116 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1","Type":"ContainerStarted","Data":"4622a1e48736adb4f3c7bca420aad649496e42d7b53155f12e54b87d7b7517d1"} Sep 18 17:16:41.443642 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:41.443148 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1","Type":"ContainerStarted","Data":"9e0a24f28b7be415829da02e2453172af6732c972ee27f82548909f920202092"} Sep 18 17:16:41.443642 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:41.443191 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1","Type":"ContainerStarted","Data":"f6173cf3146fbdb807e40743de293583f7fe320314f0a99c1f2394d75ba06861"} Sep 18 17:16:41.443642 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:41.443233 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1","Type":"ContainerStarted","Data":"e18fa0c95c44e301598ee570e2a858fd78ed9b1df931f94e800e0e63dcdbcb18"} Sep 18 17:16:41.478251 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:41.478192 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" podStartSLOduration=3.135298751 podStartE2EDuration="8.478175838s" totalImagesPullingTime="5.342877087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:41.476492388 +0000 UTC m=+114.729653230" watchObservedRunningTime="2026-09-18 17:16:41.478175838 +0000 UTC m=+114.731336694" Sep 18 17:16:41.519417 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:41.519359 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.882438975 podStartE2EDuration="10.519336179s" totalImagesPullingTime="4.61673621s" totalInitContainerRuntime="20.160994ms" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:41.517946167 +0000 UTC m=+114.771107001" watchObservedRunningTime="2026-09-18 17:16:41.519336179 +0000 UTC m=+114.772496997" Sep 18 17:16:42.030479 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:42.030446 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:42.114318 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:42.114263 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:42.447750 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:42.447720 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:42.453179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:42.453157 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f756495df-fzz5b" Sep 18 17:16:43.454559 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:43.454480 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd","Type":"ContainerStarted","Data":"4c3f4eec52371d0ecea35f41d722893acedf0ae9a7fc2a8e4942653442591108"} Sep 18 17:16:43.454898 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:43.454560 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd","Type":"ContainerStarted","Data":"66a4c9e655abfe88143a02a0f6165fc5e7d3e1d1725ae30323a2e59a44e58dc7"} Sep 18 17:16:43.454898 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:43.454601 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd","Type":"ContainerStarted","Data":"f93f7ab2af3517411d7de1d65623111778de5475fac57530ad78b6e3f77d1506"} Sep 18 17:16:43.454898 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:43.454637 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd","Type":"ContainerStarted","Data":"6b5be89fdb00d044a0879fbb1d5014fe94d1f34e18f6eb63240c9151516d5628"} Sep 18 17:16:43.454898 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:43.454664 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd","Type":"ContainerStarted","Data":"6fc65814ee15b851770637708f44ae1971d7a5dbe16ca081f05cafd75d3d8d3a"} Sep 18 17:16:43.454898 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:43.454685 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd","Type":"ContainerStarted","Data":"22653db83076c2f3eb0337a505863cea41554f21ca51894f4f331e942b20b358"} Sep 18 17:16:43.512203 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:43.512144 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.427743575 podStartE2EDuration="7.512127338s" totalImagesPullingTime="4.051798353s" totalInitContainerRuntime="32.58541ms" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:43.510884061 +0000 UTC m=+116.764044904" watchObservedRunningTime="2026-09-18 17:16:43.512127338 +0000 UTC m=+116.765288268" Sep 18 17:16:44.099793 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:44.099758 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:44.208135 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:44.208096 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:47.063896 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:47.063856 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:16:48.950456 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:48.950424 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k8tvl_e62a1939-824b-49c5-857a-c0c41fb3c31a/serve-healthcheck-canary/0.log" Sep 18 17:16:50.001640 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:50.001611 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:50.102366 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:50.102336 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:50.203769 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:50.203742 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:50.490512 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:50.490476 2560 generic.go:356] "Generic (PLEG): container finished" podID="156a0989-b40d-4b6b-84cf-c8afb86fe65c" containerID="6fea6feb612b959102fccdc5677a301cfaed3e48ded35b1a72308f8ef406b3eb" exitCode=0 Sep 18 17:16:50.490701 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:50.490555 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" event={"ID":"156a0989-b40d-4b6b-84cf-c8afb86fe65c","Type":"ContainerDied","Data":"6fea6feb612b959102fccdc5677a301cfaed3e48ded35b1a72308f8ef406b3eb"} Sep 18 17:16:50.490955 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:50.490939 2560 scope.go:117] "RemoveContainer" containerID="6fea6feb612b959102fccdc5677a301cfaed3e48ded35b1a72308f8ef406b3eb" Sep 18 17:16:51.497789 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:51.497752 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-pbzv4" event={"ID":"156a0989-b40d-4b6b-84cf-c8afb86fe65c","Type":"ContainerStarted","Data":"c96ca4d44bef3c724362c25cc167278dee1b50bc831a882e62dce403df56f132"} Sep 18 17:16:51.554635 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:51.554602 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:51.756479 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:51.756383 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:51.853878 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:51.853845 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:54.325156 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.325118 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7ff47cf8c4-czrgs" podUID="eb4ea6df-f412-483b-ad31-4dca01458ba6" containerName="console" containerID="cri-o://96d34a8adb4ed312d57c9e0ded3f5d9dfc1fcd33a47a01a84cbc89360d91f0ed" gracePeriod=15 Sep 18 17:16:54.519403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.519350 2560 generic.go:356] "Generic (PLEG): container finished" podID="eb4ea6df-f412-483b-ad31-4dca01458ba6" containerID="96d34a8adb4ed312d57c9e0ded3f5d9dfc1fcd33a47a01a84cbc89360d91f0ed" exitCode=0 Sep 18 17:16:54.519537 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.519428 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff47cf8c4-czrgs" event={"ID":"eb4ea6df-f412-483b-ad31-4dca01458ba6","Type":"ContainerDied","Data":"96d34a8adb4ed312d57c9e0ded3f5d9dfc1fcd33a47a01a84cbc89360d91f0ed"} Sep 18 17:16:54.602280 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.602202 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:54.753858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.753826 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-oauth-serving-cert\") pod \"eb4ea6df-f412-483b-ad31-4dca01458ba6\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " Sep 18 17:16:54.754025 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.753874 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-oauth-config\") pod \"eb4ea6df-f412-483b-ad31-4dca01458ba6\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " Sep 18 17:16:54.754025 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.753910 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-config\") pod \"eb4ea6df-f412-483b-ad31-4dca01458ba6\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " Sep 18 17:16:54.754025 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.753936 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-serving-cert\") pod \"eb4ea6df-f412-483b-ad31-4dca01458ba6\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " Sep 18 17:16:54.754138 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.754043 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9594t\" (UniqueName: \"kubernetes.io/projected/eb4ea6df-f412-483b-ad31-4dca01458ba6-kube-api-access-9594t\") pod \"eb4ea6df-f412-483b-ad31-4dca01458ba6\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " Sep 18 17:16:54.754138 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.754078 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-service-ca\") pod \"eb4ea6df-f412-483b-ad31-4dca01458ba6\" (UID: \"eb4ea6df-f412-483b-ad31-4dca01458ba6\") " Sep 18 17:16:54.754339 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.754276 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eb4ea6df-f412-483b-ad31-4dca01458ba6" (UID: "eb4ea6df-f412-483b-ad31-4dca01458ba6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:16:54.754564 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.754483 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-config" (OuterVolumeSpecName: "console-config") pod "eb4ea6df-f412-483b-ad31-4dca01458ba6" (UID: "eb4ea6df-f412-483b-ad31-4dca01458ba6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:16:54.754564 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.754547 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-service-ca" (OuterVolumeSpecName: "service-ca") pod "eb4ea6df-f412-483b-ad31-4dca01458ba6" (UID: "eb4ea6df-f412-483b-ad31-4dca01458ba6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:16:54.756326 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.756265 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4ea6df-f412-483b-ad31-4dca01458ba6-kube-api-access-9594t" (OuterVolumeSpecName: "kube-api-access-9594t") pod "eb4ea6df-f412-483b-ad31-4dca01458ba6" (UID: "eb4ea6df-f412-483b-ad31-4dca01458ba6"). InnerVolumeSpecName "kube-api-access-9594t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:16:54.756326 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.756282 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eb4ea6df-f412-483b-ad31-4dca01458ba6" (UID: "eb4ea6df-f412-483b-ad31-4dca01458ba6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:16:54.756481 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.756408 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eb4ea6df-f412-483b-ad31-4dca01458ba6" (UID: "eb4ea6df-f412-483b-ad31-4dca01458ba6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:16:54.855514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.855433 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9594t\" (UniqueName: \"kubernetes.io/projected/eb4ea6df-f412-483b-ad31-4dca01458ba6-kube-api-access-9594t\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:54.855514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.855463 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-service-ca\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:54.855514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.855472 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-oauth-serving-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:54.855514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.855480 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-oauth-config\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:54.855514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.855488 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-config\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:54.855514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:54.855497 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb4ea6df-f412-483b-ad31-4dca01458ba6-console-serving-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:16:55.224706 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:55.224667 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:55.224913 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:55.224748 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:16:55.525069 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:55.524997 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ff47cf8c4-czrgs" Sep 18 17:16:55.525069 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:55.525045 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ff47cf8c4-czrgs" event={"ID":"eb4ea6df-f412-483b-ad31-4dca01458ba6","Type":"ContainerDied","Data":"b11f7c365cb104742d30a419c1134cb7594e28ab4a959997cb7aca73e4e5542f"} Sep 18 17:16:55.525544 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:55.525105 2560 scope.go:117] "RemoveContainer" containerID="96d34a8adb4ed312d57c9e0ded3f5d9dfc1fcd33a47a01a84cbc89360d91f0ed" Sep 18 17:16:55.546144 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:55.546110 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ff47cf8c4-czrgs"] Sep 18 17:16:55.552516 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:55.552489 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7ff47cf8c4-czrgs"] Sep 18 17:16:56.863507 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:56.863479 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb4ea6df-f412-483b-ad31-4dca01458ba6" path="/var/lib/kubelet/pods/eb4ea6df-f412-483b-ad31-4dca01458ba6/volumes" Sep 18 17:16:57.018787 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:57.018755 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:57.120195 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:57.120117 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:58.588718 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:58.588670 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:16:58.591031 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:58.591002 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd4d7977-637e-4068-8d44-515534534834-metrics-certs\") pod \"network-metrics-daemon-clj2d\" (UID: \"cd4d7977-637e-4068-8d44-515534534834\") " pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:16:58.779372 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:58.779342 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2h6n2\"" Sep 18 17:16:58.786774 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:58.786748 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-clj2d" Sep 18 17:16:59.070326 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:59.070280 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-clj2d"] Sep 18 17:16:59.073606 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:16:59.073578 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4d7977_637e_4068_8d44_515534534834.slice/crio-21a3d007125af6eabde7af53eaf7ff2bc1cf67232071c4f9126db575952ae398 WatchSource:0}: Error finding container 21a3d007125af6eabde7af53eaf7ff2bc1cf67232071c4f9126db575952ae398: Status 404 returned error can't find the container with id 21a3d007125af6eabde7af53eaf7ff2bc1cf67232071c4f9126db575952ae398 Sep 18 17:16:59.535650 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:59.535619 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:16:59.543168 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:59.543134 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-clj2d" event={"ID":"cd4d7977-637e-4068-8d44-515534534834","Type":"ContainerStarted","Data":"21a3d007125af6eabde7af53eaf7ff2bc1cf67232071c4f9126db575952ae398"} Sep 18 17:16:59.624691 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:16:59.624654 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:17:01.559258 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:01.559209 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-clj2d" event={"ID":"cd4d7977-637e-4068-8d44-515534534834","Type":"ContainerStarted","Data":"9f3116f2344f23f6b9c1ebf10d0bdef03b08c196e40c915ac58738d24de16e8a"} Sep 18 17:17:01.559762 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:01.559304 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-clj2d" event={"ID":"cd4d7977-637e-4068-8d44-515534534834","Type":"ContainerStarted","Data":"e13718097b80d94579c940dfad268a9e99dfca3350d9ef308eb134d34eec70b1"} Sep 18 17:17:01.583254 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:01.583200 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-clj2d" podStartSLOduration=131.962035473 podStartE2EDuration="2m13.583181966s" totalImagesPullingTime="1.621146493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:17:01.580726316 +0000 UTC m=+134.833887153" watchObservedRunningTime="2026-09-18 17:17:01.583181966 +0000 UTC m=+134.836342782" Sep 18 17:17:01.706476 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:01.706431 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:17:01.785485 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:01.785448 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:17:15.229659 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:15.229630 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:17:15.233458 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:15.233436 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5d4c6d666-2lpbp" Sep 18 17:17:37.063544 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:37.063454 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:17:37.082527 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:37.082506 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:17:37.732546 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:37.732512 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 18 17:17:46.614629 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:46.614588 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:17:46.714439 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:46.714406 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:17:50.756867 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:50.756824 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:17:50.840264 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:50.840226 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:17:50.954223 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:50.954194 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:17:51.697061 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:51.697026 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:17:51.874896 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:51.874861 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:17:58.428117 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:58.428085 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:17:58.505829 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:17:58.505798 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:18:01.767021 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:18:01.766989 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:18:01.876018 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:18:01.875972 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:18:01.945487 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:18:01.945457 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:18:12.028622 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:18:12.028592 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:18:12.124459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:18:12.124428 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:18:19.079379 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:18:19.079347 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:18:19.148375 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:18:19.148342 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:18:21.902330 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:18:21.902275 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:18:55.307126 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:18:55.307095 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:18:55.369133 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:18:55.369103 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:18:55.733565 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:18:55.733534 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:18:55.844356 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:18:55.844324 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:18.054284 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:18.054251 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:18.138126 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:18.138096 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:22.059810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:22.059774 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:22.157803 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:22.157764 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:24.107288 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:24.107257 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:24.266057 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:24.266021 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:24.383957 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:24.383874 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:26.353251 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:26.353223 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:27.029743 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:27.029716 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:34.978017 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:34.977960 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:37.709050 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:37.709016 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:37.836122 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:37.836086 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:45.125543 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:45.125507 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:45.226492 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:45.226460 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:48.782267 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:48.782241 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:19:48.784193 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:48.784170 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:19:48.788805 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:48.788787 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:19:48.790754 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:48.790738 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:19:48.792822 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:48.792804 2560 kubelet.go:1648] "Image garbage collection succeeded" Sep 18 17:19:49.753010 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:49.752983 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:49.849626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:49.849594 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:19:51.814834 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:19:51.814805 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:07.318171 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:07.318136 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:08.556380 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.556350 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj"] Sep 18 17:20:08.556755 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.556700 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb4ea6df-f412-483b-ad31-4dca01458ba6" containerName="console" Sep 18 17:20:08.556755 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.556710 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb4ea6df-f412-483b-ad31-4dca01458ba6" containerName="console" Sep 18 17:20:08.556825 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.556775 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb4ea6df-f412-483b-ad31-4dca01458ba6" containerName="console" Sep 18 17:20:08.559122 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.559102 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.561991 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.561971 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 18 17:20:08.562317 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.562285 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 18 17:20:08.562317 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.562286 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dlwd7\"" Sep 18 17:20:08.562592 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.562579 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 18 17:20:08.563632 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.563615 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 18 17:20:08.563717 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.563645 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppknh\" (UniqueName: \"kubernetes.io/projected/9ecd4364-2a82-4806-8853-a6d9240e06c6-kube-api-access-ppknh\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.563717 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.563676 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ecd4364-2a82-4806-8853-a6d9240e06c6-manifests\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.563717 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.563693 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ecd4364-2a82-4806-8853-a6d9240e06c6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.563867 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.563725 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecd4364-2a82-4806-8853-a6d9240e06c6-tmp\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.563867 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.563747 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ecd4364-2a82-4806-8853-a6d9240e06c6-webhook-cert\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.590144 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.590115 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj"] Sep 18 17:20:08.664159 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.664128 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppknh\" (UniqueName: \"kubernetes.io/projected/9ecd4364-2a82-4806-8853-a6d9240e06c6-kube-api-access-ppknh\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.664159 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.664165 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ecd4364-2a82-4806-8853-a6d9240e06c6-manifests\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.664390 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.664187 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ecd4364-2a82-4806-8853-a6d9240e06c6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.664390 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.664226 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecd4364-2a82-4806-8853-a6d9240e06c6-tmp\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.664390 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.664251 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ecd4364-2a82-4806-8853-a6d9240e06c6-webhook-cert\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.664614 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.664593 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ecd4364-2a82-4806-8853-a6d9240e06c6-manifests\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.667062 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.667033 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecd4364-2a82-4806-8853-a6d9240e06c6-tmp\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.667202 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.667171 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ecd4364-2a82-4806-8853-a6d9240e06c6-webhook-cert\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.667349 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.667330 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ecd4364-2a82-4806-8853-a6d9240e06c6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.673433 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.673411 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppknh\" (UniqueName: \"kubernetes.io/projected/9ecd4364-2a82-4806-8853-a6d9240e06c6-kube-api-access-ppknh\") pod \"opendatahub-operator-controller-manager-9496889b9-g6qzj\" (UID: \"9ecd4364-2a82-4806-8853-a6d9240e06c6\") " pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:08.869613 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:08.869526 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:09.140691 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:09.140618 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:09.145809 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:09.145786 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj"] Sep 18 17:20:09.147471 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:20:09.147444 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecd4364_2a82_4806_8853_a6d9240e06c6.slice/crio-35907ccdb98d89642655c0ef85138666688aeb52847b3dd7d5f42a0fee97e3d6 WatchSource:0}: Error finding container 35907ccdb98d89642655c0ef85138666688aeb52847b3dd7d5f42a0fee97e3d6: Status 404 returned error can't find the container with id 35907ccdb98d89642655c0ef85138666688aeb52847b3dd7d5f42a0fee97e3d6 Sep 18 17:20:09.149937 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:09.149921 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 17:20:09.360635 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:09.360601 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" event={"ID":"9ecd4364-2a82-4806-8853-a6d9240e06c6","Type":"ContainerStarted","Data":"35907ccdb98d89642655c0ef85138666688aeb52847b3dd7d5f42a0fee97e3d6"} Sep 18 17:20:12.376880 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:12.376801 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" event={"ID":"9ecd4364-2a82-4806-8853-a6d9240e06c6","Type":"ContainerStarted","Data":"0ea05ed6975d84ec2d9ef5547ba3a01f492ec01e289b3b1a54ff3de1e9df3a7f"} Sep 18 17:20:13.383438 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:13.383405 2560 generic.go:356] "Generic (PLEG): container finished" podID="9ecd4364-2a82-4806-8853-a6d9240e06c6" containerID="0ea05ed6975d84ec2d9ef5547ba3a01f492ec01e289b3b1a54ff3de1e9df3a7f" exitCode=0 Sep 18 17:20:13.383841 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:13.383491 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" event={"ID":"9ecd4364-2a82-4806-8853-a6d9240e06c6","Type":"ContainerDied","Data":"0ea05ed6975d84ec2d9ef5547ba3a01f492ec01e289b3b1a54ff3de1e9df3a7f"} Sep 18 17:20:14.390093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:14.390059 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" event={"ID":"9ecd4364-2a82-4806-8853-a6d9240e06c6","Type":"ContainerStarted","Data":"fa99bd70d13f11848310b5f4546548721e9180dd6d59fea9faaafe8908c81ea9"} Sep 18 17:20:14.390477 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:14.390169 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:14.422885 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:14.422843 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" podStartSLOduration=2.160678006 podStartE2EDuration="6.4228264s" totalImagesPullingTime="3.236560987s" totalInitContainerRuntime="1.025587407s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:20:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:20:14.42124944 +0000 UTC m=+327.674410290" watchObservedRunningTime="2026-09-18 17:20:14.4228264 +0000 UTC m=+327.675987218" Sep 18 17:20:16.043747 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:16.043713 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:16.139175 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:16.139152 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:19.833494 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:19.833462 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:25.397449 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:25.397416 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9496889b9-g6qzj" Sep 18 17:20:28.901494 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:28.901460 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:29.046564 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:29.046525 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:29.316502 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:29.316425 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:29.503899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:29.503862 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:31.345974 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:31.345897 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:31.464277 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:31.464245 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:32.914464 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:32.914431 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:36.645218 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:36.645182 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:36.747111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:36.747075 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:36.953348 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:36.953320 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:37.003857 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.003826 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj"] Sep 18 17:20:37.007035 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.007014 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.010481 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.010449 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 18 17:20:37.010610 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.010493 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 18 17:20:37.010679 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.010606 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 18 17:20:37.010736 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.010698 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-z4tlw\"" Sep 18 17:20:37.024874 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.024846 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj"] Sep 18 17:20:37.102982 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.102952 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:37.114164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.114125 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.114328 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.114213 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwqb8\" (UniqueName: \"kubernetes.io/projected/96c39c3a-28d7-483f-9e54-6ba0eec7b333-kube-api-access-vwqb8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.114328 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.114251 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.114328 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.114279 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.114494 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.114329 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.114494 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.114356 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.114494 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.114397 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.114494 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.114435 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.114494 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.114462 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.215512 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.215422 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.215512 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.215461 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.215512 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.215490 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.215753 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.215552 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwqb8\" (UniqueName: \"kubernetes.io/projected/96c39c3a-28d7-483f-9e54-6ba0eec7b333-kube-api-access-vwqb8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.215753 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.215586 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.215753 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.215687 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.215753 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.215717 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.215753 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.215746 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.215997 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.215791 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.215997 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.215875 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.215997 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.215946 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.216157 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.216087 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.216264 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.216244 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.216353 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.216248 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.217878 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.217862 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.218287 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.218266 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.225173 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.225148 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwqb8\" (UniqueName: \"kubernetes.io/projected/96c39c3a-28d7-483f-9e54-6ba0eec7b333-kube-api-access-vwqb8\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.227338 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.227320 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c39c3a-28d7-483f-9e54-6ba0eec7b333-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj\" (UID: \"96c39c3a-28d7-483f-9e54-6ba0eec7b333\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.316639 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.316610 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:37.317633 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.317609 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:37.607485 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:37.607461 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj"] Sep 18 17:20:37.609504 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:20:37.609477 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c39c3a_28d7_483f_9e54_6ba0eec7b333.slice/crio-ec1d00504ede8af87c25be0c81afd229a0d79eef74a5bc0b10b36c7900804d30 WatchSource:0}: Error finding container ec1d00504ede8af87c25be0c81afd229a0d79eef74a5bc0b10b36c7900804d30: Status 404 returned error can't find the container with id ec1d00504ede8af87c25be0c81afd229a0d79eef74a5bc0b10b36c7900804d30 Sep 18 17:20:38.507655 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:38.507616 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" event={"ID":"96c39c3a-28d7-483f-9e54-6ba0eec7b333","Type":"ContainerStarted","Data":"ec1d00504ede8af87c25be0c81afd229a0d79eef74a5bc0b10b36c7900804d30"} Sep 18 17:20:40.034149 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:40.034020 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:40.129566 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:40.129537 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:40.230672 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:40.230636 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:40.614729 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:40.614693 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:40.716605 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:40.716575 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:40.783398 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:40.783361 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:42.386434 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:42.386384 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:20:42.386726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:42.386467 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:20:42.386726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:42.386499 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:20:42.527584 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:42.527544 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" event={"ID":"96c39c3a-28d7-483f-9e54-6ba0eec7b333","Type":"ContainerStarted","Data":"e98ce1d3e7130ee47a116243b37f714e50f941cb3be06ac030361c9c385d5f08"} Sep 18 17:20:42.562650 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:42.562592 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" podStartSLOduration=1.7879749569999999 podStartE2EDuration="6.562574181s" totalImagesPullingTime="4.774599224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:20:42.561410455 +0000 UTC m=+355.814571295" watchObservedRunningTime="2026-09-18 17:20:42.562574181 +0000 UTC m=+355.815734997" Sep 18 17:20:43.316874 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:43.316836 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:43.321412 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:43.321386 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:43.532370 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:43.532342 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:43.533278 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:43.533256 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj" Sep 18 17:20:48.478836 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.478804 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k"] Sep 18 17:20:48.482212 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.482190 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:48.485006 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.484983 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 18 17:20:48.486211 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.486187 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 18 17:20:48.486393 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.486311 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 18 17:20:48.486483 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.486346 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 18 17:20:48.486483 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.486404 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qb9bx\"" Sep 18 17:20:48.486483 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.486439 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 18 17:20:48.494799 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.494774 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k"] Sep 18 17:20:48.621154 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.621115 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/04323ef3-f6a8-4414-af55-9ddc73a25ab4-metrics-cert\") pod \"lws-controller-manager-5b67bbc446-dmb2k\" (UID: \"04323ef3-f6a8-4414-af55-9ddc73a25ab4\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:48.621380 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.621178 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bqmg\" (UniqueName: \"kubernetes.io/projected/04323ef3-f6a8-4414-af55-9ddc73a25ab4-kube-api-access-5bqmg\") pod \"lws-controller-manager-5b67bbc446-dmb2k\" (UID: \"04323ef3-f6a8-4414-af55-9ddc73a25ab4\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:48.621380 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.621332 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/04323ef3-f6a8-4414-af55-9ddc73a25ab4-manager-config\") pod \"lws-controller-manager-5b67bbc446-dmb2k\" (UID: \"04323ef3-f6a8-4414-af55-9ddc73a25ab4\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:48.621462 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.621407 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04323ef3-f6a8-4414-af55-9ddc73a25ab4-cert\") pod \"lws-controller-manager-5b67bbc446-dmb2k\" (UID: \"04323ef3-f6a8-4414-af55-9ddc73a25ab4\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:48.722624 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.722581 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04323ef3-f6a8-4414-af55-9ddc73a25ab4-cert\") pod \"lws-controller-manager-5b67bbc446-dmb2k\" (UID: \"04323ef3-f6a8-4414-af55-9ddc73a25ab4\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:48.722624 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.722627 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/04323ef3-f6a8-4414-af55-9ddc73a25ab4-metrics-cert\") pod \"lws-controller-manager-5b67bbc446-dmb2k\" (UID: \"04323ef3-f6a8-4414-af55-9ddc73a25ab4\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:48.722868 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.722677 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bqmg\" (UniqueName: \"kubernetes.io/projected/04323ef3-f6a8-4414-af55-9ddc73a25ab4-kube-api-access-5bqmg\") pod \"lws-controller-manager-5b67bbc446-dmb2k\" (UID: \"04323ef3-f6a8-4414-af55-9ddc73a25ab4\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:48.722934 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.722911 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/04323ef3-f6a8-4414-af55-9ddc73a25ab4-manager-config\") pod \"lws-controller-manager-5b67bbc446-dmb2k\" (UID: \"04323ef3-f6a8-4414-af55-9ddc73a25ab4\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:48.723538 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.723514 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/04323ef3-f6a8-4414-af55-9ddc73a25ab4-manager-config\") pod \"lws-controller-manager-5b67bbc446-dmb2k\" (UID: \"04323ef3-f6a8-4414-af55-9ddc73a25ab4\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:48.725388 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.725359 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04323ef3-f6a8-4414-af55-9ddc73a25ab4-cert\") pod \"lws-controller-manager-5b67bbc446-dmb2k\" (UID: \"04323ef3-f6a8-4414-af55-9ddc73a25ab4\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:48.725388 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.725368 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/04323ef3-f6a8-4414-af55-9ddc73a25ab4-metrics-cert\") pod \"lws-controller-manager-5b67bbc446-dmb2k\" (UID: \"04323ef3-f6a8-4414-af55-9ddc73a25ab4\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:48.738664 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.738598 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bqmg\" (UniqueName: \"kubernetes.io/projected/04323ef3-f6a8-4414-af55-9ddc73a25ab4-kube-api-access-5bqmg\") pod \"lws-controller-manager-5b67bbc446-dmb2k\" (UID: \"04323ef3-f6a8-4414-af55-9ddc73a25ab4\") " pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:48.795690 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.795665 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qb9bx\"" Sep 18 17:20:48.802772 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:48.802749 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:49.082589 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:49.082561 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k"] Sep 18 17:20:49.084120 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:20:49.084094 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04323ef3_f6a8_4414_af55_9ddc73a25ab4.slice/crio-55217a7bdc517f456c48c6165067c60d735aefcab2386a85a6a7c71000dd5231 WatchSource:0}: Error finding container 55217a7bdc517f456c48c6165067c60d735aefcab2386a85a6a7c71000dd5231: Status 404 returned error can't find the container with id 55217a7bdc517f456c48c6165067c60d735aefcab2386a85a6a7c71000dd5231 Sep 18 17:20:49.563493 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:49.563459 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" event={"ID":"04323ef3-f6a8-4414-af55-9ddc73a25ab4","Type":"ContainerStarted","Data":"55217a7bdc517f456c48c6165067c60d735aefcab2386a85a6a7c71000dd5231"} Sep 18 17:20:50.436760 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.436548 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:50.550752 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.550702 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:50.559971 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.559940 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f"] Sep 18 17:20:50.563654 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.563630 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f" Sep 18 17:20:50.570223 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.570198 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 18 17:20:50.570385 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.570269 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pprkx\"" Sep 18 17:20:50.631555 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.631521 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f"] Sep 18 17:20:50.640486 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.640457 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt"] Sep 18 17:20:50.645645 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.645619 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" Sep 18 17:20:50.651018 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.650997 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 18 17:20:50.657351 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.657327 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7t574\"" Sep 18 17:20:50.662852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.662817 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt"] Sep 18 17:20:50.699781 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.699695 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:50.743521 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.743489 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/281a8253-e792-4359-8464-823a81f61dc0-cert\") pod \"kserve-controller-manager-d6f4b65b8-zzn5f\" (UID: \"281a8253-e792-4359-8464-823a81f61dc0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f" Sep 18 17:20:50.743721 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.743573 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z69fk\" (UniqueName: \"kubernetes.io/projected/281a8253-e792-4359-8464-823a81f61dc0-kube-api-access-z69fk\") pod \"kserve-controller-manager-d6f4b65b8-zzn5f\" (UID: \"281a8253-e792-4359-8464-823a81f61dc0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f" Sep 18 17:20:50.844997 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.844949 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/281a8253-e792-4359-8464-823a81f61dc0-cert\") pod \"kserve-controller-manager-d6f4b65b8-zzn5f\" (UID: \"281a8253-e792-4359-8464-823a81f61dc0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f" Sep 18 17:20:50.845175 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.845015 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkxqz\" (UniqueName: \"kubernetes.io/projected/c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d-kube-api-access-mkxqz\") pod \"llmisvc-controller-manager-777c6bd7d5-jhblt\" (UID: \"c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" Sep 18 17:20:50.845175 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.845069 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jhblt\" (UID: \"c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" Sep 18 17:20:50.845175 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.845159 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z69fk\" (UniqueName: \"kubernetes.io/projected/281a8253-e792-4359-8464-823a81f61dc0-kube-api-access-z69fk\") pod \"kserve-controller-manager-d6f4b65b8-zzn5f\" (UID: \"281a8253-e792-4359-8464-823a81f61dc0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f" Sep 18 17:20:50.848321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.847962 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/281a8253-e792-4359-8464-823a81f61dc0-cert\") pod \"kserve-controller-manager-d6f4b65b8-zzn5f\" (UID: \"281a8253-e792-4359-8464-823a81f61dc0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f" Sep 18 17:20:50.861922 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.861891 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z69fk\" (UniqueName: \"kubernetes.io/projected/281a8253-e792-4359-8464-823a81f61dc0-kube-api-access-z69fk\") pod \"kserve-controller-manager-d6f4b65b8-zzn5f\" (UID: \"281a8253-e792-4359-8464-823a81f61dc0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f" Sep 18 17:20:50.876513 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.876482 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f" Sep 18 17:20:50.877283 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.877253 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:50.946156 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.946114 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkxqz\" (UniqueName: \"kubernetes.io/projected/c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d-kube-api-access-mkxqz\") pod \"llmisvc-controller-manager-777c6bd7d5-jhblt\" (UID: \"c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" Sep 18 17:20:50.946353 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.946178 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jhblt\" (UID: \"c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" Sep 18 17:20:50.946353 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:20:50.946303 2560 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 18 17:20:50.946485 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:20:50.946382 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d-cert podName:c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d nodeName:}" failed. No retries permitted until 2026-09-18 17:20:51.446363858 +0000 UTC m=+364.699524663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d-cert") pod "llmisvc-controller-manager-777c6bd7d5-jhblt" (UID: "c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d") : secret "llmisvc-webhook-server-cert" not found Sep 18 17:20:50.961473 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.961391 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkxqz\" (UniqueName: \"kubernetes.io/projected/c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d-kube-api-access-mkxqz\") pod \"llmisvc-controller-manager-777c6bd7d5-jhblt\" (UID: \"c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" Sep 18 17:20:50.975500 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:50.975470 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:51.079024 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:51.078984 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:51.203823 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:51.203792 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:20:51.451676 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:51.451640 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jhblt\" (UID: \"c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" Sep 18 17:20:51.454408 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:51.454379 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jhblt\" (UID: \"c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" Sep 18 17:20:51.559200 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:51.559160 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" Sep 18 17:20:51.936304 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:51.936254 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f"] Sep 18 17:20:51.940398 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:20:51.940367 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281a8253_e792_4359_8464_823a81f61dc0.slice/crio-7ad4ec8daa7e16cc0a4680b48367f90fa03430e01688623e7f13a6b83fbacb5c WatchSource:0}: Error finding container 7ad4ec8daa7e16cc0a4680b48367f90fa03430e01688623e7f13a6b83fbacb5c: Status 404 returned error can't find the container with id 7ad4ec8daa7e16cc0a4680b48367f90fa03430e01688623e7f13a6b83fbacb5c Sep 18 17:20:51.940880 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:51.940851 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt"] Sep 18 17:20:51.942283 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:20:51.942256 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc69e0dab_9ad4_4d1c_9fdf_4c7fc0e28d5d.slice/crio-2c32aa06167e4fcae918d98a1dc6a655e44259a6f3ee83c8b64ea090474eadac WatchSource:0}: Error finding container 2c32aa06167e4fcae918d98a1dc6a655e44259a6f3ee83c8b64ea090474eadac: Status 404 returned error can't find the container with id 2c32aa06167e4fcae918d98a1dc6a655e44259a6f3ee83c8b64ea090474eadac Sep 18 17:20:52.582386 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:52.582354 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f" event={"ID":"281a8253-e792-4359-8464-823a81f61dc0","Type":"ContainerStarted","Data":"7ad4ec8daa7e16cc0a4680b48367f90fa03430e01688623e7f13a6b83fbacb5c"} Sep 18 17:20:52.584047 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:52.583998 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" event={"ID":"c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d","Type":"ContainerStarted","Data":"2c32aa06167e4fcae918d98a1dc6a655e44259a6f3ee83c8b64ea090474eadac"} Sep 18 17:20:52.585841 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:52.585818 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" event={"ID":"04323ef3-f6a8-4414-af55-9ddc73a25ab4","Type":"ContainerStarted","Data":"81044cc05f4e6b2b2e38f2f09c29c79e1810b4cf93efac91f57a1e47e5b6b823"} Sep 18 17:20:52.585971 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:52.585956 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:20:55.605670 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:55.605639 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f" event={"ID":"281a8253-e792-4359-8464-823a81f61dc0","Type":"ContainerStarted","Data":"d1f7cb40c8cea920f67edaafcbb6877391434abf58ee723590b5e88eac9916a9"} Sep 18 17:20:55.606024 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:55.605702 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f" Sep 18 17:20:55.626908 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:55.626866 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f" podStartSLOduration=2.064989396 podStartE2EDuration="5.626851595s" totalImagesPullingTime="3.561862199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:20:55.624423148 +0000 UTC m=+368.877583978" watchObservedRunningTime="2026-09-18 17:20:55.626851595 +0000 UTC m=+368.880012441" Sep 18 17:20:55.627267 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:55.627247 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" podStartSLOduration=4.982151625 podStartE2EDuration="7.627240925s" totalImagesPullingTime="2.6450893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:20:52.621633911 +0000 UTC m=+365.874794754" watchObservedRunningTime="2026-09-18 17:20:55.627240925 +0000 UTC m=+368.880401745" Sep 18 17:20:56.612187 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:56.612154 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" event={"ID":"c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d","Type":"ContainerStarted","Data":"66d47d967da01b663ad516d776ece2fbe2629d399319e1e0b80efa6f7405f76e"} Sep 18 17:20:56.612573 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:56.612245 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" Sep 18 17:20:56.632549 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:20:56.632482 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" podStartSLOduration=3.073209086 podStartE2EDuration="6.632468774s" totalImagesPullingTime="3.559259688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:20:56.631636884 +0000 UTC m=+369.884797717" watchObservedRunningTime="2026-09-18 17:20:56.632468774 +0000 UTC m=+369.885629590" Sep 18 17:21:03.596577 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:03.596550 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b67bbc446-dmb2k" Sep 18 17:21:25.392219 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.392187 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65c9bfcd84-wv27g"] Sep 18 17:21:25.459853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.459822 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:21:25.505779 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.505737 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c9bfcd84-wv27g"] Sep 18 17:21:25.506211 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.506181 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.509763 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.509743 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 18 17:21:25.509882 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.509743 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 18 17:21:25.509882 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.509835 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 18 17:21:25.510032 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.510018 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 18 17:21:25.511019 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.510998 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kp6ft\"" Sep 18 17:21:25.512035 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.512018 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 18 17:21:25.517016 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.516999 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 18 17:21:25.540492 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.540472 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:21:25.657072 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.657046 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-trusted-ca-bundle\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.657222 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.657079 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jz8r\" (UniqueName: \"kubernetes.io/projected/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-kube-api-access-2jz8r\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.657222 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.657100 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-console-config\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.657382 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.657233 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-console-oauth-config\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.657382 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.657266 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-service-ca\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.657382 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.657333 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-console-serving-cert\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.657382 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.657365 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-oauth-serving-cert\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.758596 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.758563 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-trusted-ca-bundle\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.758596 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.758596 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jz8r\" (UniqueName: \"kubernetes.io/projected/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-kube-api-access-2jz8r\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.758812 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.758616 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-console-config\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.758812 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.758660 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-console-oauth-config\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.758812 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.758674 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-service-ca\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.758812 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.758697 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-console-serving-cert\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.758812 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.758711 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-oauth-serving-cert\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.759454 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.759428 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-console-config\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.759579 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.759535 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-trusted-ca-bundle\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.759579 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.759554 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-service-ca\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.759665 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.759592 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-oauth-serving-cert\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.761306 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.761265 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-console-serving-cert\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.761406 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.761309 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-console-oauth-config\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.767043 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.767020 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jz8r\" (UniqueName: \"kubernetes.io/projected/4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd-kube-api-access-2jz8r\") pod \"console-65c9bfcd84-wv27g\" (UID: \"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd\") " pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:25.817072 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:25.817047 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:26.112212 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:26.112185 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c9bfcd84-wv27g"] Sep 18 17:21:26.114102 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:21:26.114078 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1d5fe1_b9d0_4a77_8ec1_63e5061b2dcd.slice/crio-6e53d4f19e9128d09c25b25199c345b859e62df4d12fb80775d3017526ee27ab WatchSource:0}: Error finding container 6e53d4f19e9128d09c25b25199c345b859e62df4d12fb80775d3017526ee27ab: Status 404 returned error can't find the container with id 6e53d4f19e9128d09c25b25199c345b859e62df4d12fb80775d3017526ee27ab Sep 18 17:21:26.617804 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:26.617768 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zzn5f" Sep 18 17:21:26.753112 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:26.753073 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c9bfcd84-wv27g" event={"ID":"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd","Type":"ContainerStarted","Data":"e8905b5c9df058447aeccb09d4f3e98ab8d37695e7c0776050b2dff59ed52de4"} Sep 18 17:21:26.753325 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:26.753128 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c9bfcd84-wv27g" event={"ID":"4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd","Type":"ContainerStarted","Data":"6e53d4f19e9128d09c25b25199c345b859e62df4d12fb80775d3017526ee27ab"} Sep 18 17:21:26.775698 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:26.775646 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65c9bfcd84-wv27g" podStartSLOduration=1.7756254070000002 podStartE2EDuration="1.775625407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:21:26.774674937 +0000 UTC m=+400.027835774" watchObservedRunningTime="2026-09-18 17:21:26.775625407 +0000 UTC m=+400.028786227" Sep 18 17:21:27.618710 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:27.618683 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jhblt" Sep 18 17:21:35.818076 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:35.818043 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:35.818588 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:35.818099 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:35.822954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:35.822933 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:36.800906 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:36.800879 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65c9bfcd84-wv27g" Sep 18 17:21:37.022570 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:37.022535 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:21:37.124524 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:37.124446 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:21:37.198231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:37.198183 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:21:40.710412 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:40.710375 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:21:41.001647 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:41.001562 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:21:41.101464 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:41.101417 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:21:54.976155 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:54.976118 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:21:56.932265 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:56.932234 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:21:57.031130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:57.031094 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:21:58.985731 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:58.985702 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:21:59.784889 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:21:59.784858 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:02.023421 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:02.023385 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:02.138663 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:02.138633 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:02.248244 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:02.248208 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:03.702406 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:03.702371 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2"] Sep 18 17:22:03.705681 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:03.705660 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:03.716191 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:03.716167 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 18 17:22:03.716473 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:03.716457 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gh7lc\"" Sep 18 17:22:03.717186 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:03.717173 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 18 17:22:03.726495 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:03.726463 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2"] Sep 18 17:22:03.825158 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:03.825120 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8q67\" (UniqueName: \"kubernetes.io/projected/e3ff6443-cd2f-4900-b5b1-4be46b03dc91-kube-api-access-z8q67\") pod \"limitador-operator-controller-manager-74495c69f8-77zb2\" (UID: \"e3ff6443-cd2f-4900-b5b1-4be46b03dc91\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:03.926056 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:03.926025 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8q67\" (UniqueName: \"kubernetes.io/projected/e3ff6443-cd2f-4900-b5b1-4be46b03dc91-kube-api-access-z8q67\") pod \"limitador-operator-controller-manager-74495c69f8-77zb2\" (UID: \"e3ff6443-cd2f-4900-b5b1-4be46b03dc91\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:03.940995 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:03.940969 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8q67\" (UniqueName: \"kubernetes.io/projected/e3ff6443-cd2f-4900-b5b1-4be46b03dc91-kube-api-access-z8q67\") pod \"limitador-operator-controller-manager-74495c69f8-77zb2\" (UID: \"e3ff6443-cd2f-4900-b5b1-4be46b03dc91\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:04.016529 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:04.016441 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:04.094734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:04.094700 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:04.308794 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:04.308764 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2"] Sep 18 17:22:04.310836 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:22:04.310811 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ff6443_cd2f_4900_b5b1_4be46b03dc91.slice/crio-7ff627eac78924fe50aa21b3fb96a0d2ad0382ae79f64f3bc50c75f8d45f58eb WatchSource:0}: Error finding container 7ff627eac78924fe50aa21b3fb96a0d2ad0382ae79f64f3bc50c75f8d45f58eb: Status 404 returned error can't find the container with id 7ff627eac78924fe50aa21b3fb96a0d2ad0382ae79f64f3bc50c75f8d45f58eb Sep 18 17:22:04.553565 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:04.553536 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:04.806689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:04.806654 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:04.929189 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:04.929160 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" event={"ID":"e3ff6443-cd2f-4900-b5b1-4be46b03dc91","Type":"ContainerStarted","Data":"7ff627eac78924fe50aa21b3fb96a0d2ad0382ae79f64f3bc50c75f8d45f58eb"} Sep 18 17:22:04.962594 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:04.962526 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:05.108048 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:05.107966 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:05.333681 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:05.333641 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:05.431119 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:05.431055 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:07.947914 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:07.947877 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" event={"ID":"e3ff6443-cd2f-4900-b5b1-4be46b03dc91","Type":"ContainerStarted","Data":"6bfd1e9db2b96c9be829152d04b485ca740b4947fef909f1d5bd34c1684ed55b"} Sep 18 17:22:07.948384 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:07.947938 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:07.970631 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:07.970579 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" podStartSLOduration=2.36513659 podStartE2EDuration="4.970562608s" totalImagesPullingTime="2.605426018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:22:07.969439735 +0000 UTC m=+441.222600575" watchObservedRunningTime="2026-09-18 17:22:07.970562608 +0000 UTC m=+441.223723421" Sep 18 17:22:08.761514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:08.761469 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:09.260356 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:09.260288 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:11.088412 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:11.088380 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lkjhw"] Sep 18 17:22:11.092045 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:11.092029 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lkjhw" Sep 18 17:22:11.095127 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:11.095103 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 18 17:22:11.095996 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:11.095980 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5csx5\"" Sep 18 17:22:11.104599 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:11.104578 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lkjhw"] Sep 18 17:22:11.199191 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:11.199157 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vdv\" (UniqueName: \"kubernetes.io/projected/7d3e0259-5dea-4114-a8ab-f4206d0d260c-kube-api-access-47vdv\") pod \"dns-operator-controller-manager-7b84cd7c9c-lkjhw\" (UID: \"7d3e0259-5dea-4114-a8ab-f4206d0d260c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lkjhw" Sep 18 17:22:11.299893 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:11.299853 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vdv\" (UniqueName: \"kubernetes.io/projected/7d3e0259-5dea-4114-a8ab-f4206d0d260c-kube-api-access-47vdv\") pod \"dns-operator-controller-manager-7b84cd7c9c-lkjhw\" (UID: \"7d3e0259-5dea-4114-a8ab-f4206d0d260c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lkjhw" Sep 18 17:22:11.314779 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:11.314743 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vdv\" (UniqueName: \"kubernetes.io/projected/7d3e0259-5dea-4114-a8ab-f4206d0d260c-kube-api-access-47vdv\") pod \"dns-operator-controller-manager-7b84cd7c9c-lkjhw\" (UID: \"7d3e0259-5dea-4114-a8ab-f4206d0d260c\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lkjhw" Sep 18 17:22:11.402815 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:11.402778 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lkjhw" Sep 18 17:22:11.692807 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:11.692781 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lkjhw"] Sep 18 17:22:11.694748 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:22:11.694720 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3e0259_5dea_4114_a8ab_f4206d0d260c.slice/crio-16b985c32ebd302ea818af82604f3160fa2dacebbed7951612246b95e2fe7857 WatchSource:0}: Error finding container 16b985c32ebd302ea818af82604f3160fa2dacebbed7951612246b95e2fe7857: Status 404 returned error can't find the container with id 16b985c32ebd302ea818af82604f3160fa2dacebbed7951612246b95e2fe7857 Sep 18 17:22:11.972130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:11.972049 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lkjhw" event={"ID":"7d3e0259-5dea-4114-a8ab-f4206d0d260c","Type":"ContainerStarted","Data":"16b985c32ebd302ea818af82604f3160fa2dacebbed7951612246b95e2fe7857"} Sep 18 17:22:13.159021 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:13.158982 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:13.240220 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:13.240191 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:13.750132 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:13.750100 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:13.850781 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:13.850751 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:13.984941 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:13.984904 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lkjhw" event={"ID":"7d3e0259-5dea-4114-a8ab-f4206d0d260c","Type":"ContainerStarted","Data":"863ee44d0e45574a5e102fb65806fb82516311aef4d46267b413a74c65d24446"} Sep 18 17:22:13.985155 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:13.985052 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lkjhw" Sep 18 17:22:14.015479 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:14.015376 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lkjhw" podStartSLOduration=1.697331482 podStartE2EDuration="3.015359291s" totalImagesPullingTime="1.318027809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:22:14.014186867 +0000 UTC m=+447.267347717" watchObservedRunningTime="2026-09-18 17:22:14.015359291 +0000 UTC m=+447.268520107" Sep 18 17:22:15.232881 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:15.232852 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:15.491836 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:15.491755 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:17.631540 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:17.631502 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:17.858117 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:17.858006 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:18.957548 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:18.957525 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:22.599928 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:22.599892 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:22.760772 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:22.760733 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:23.183659 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:23.183633 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:23.292183 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:23.292147 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:23.870228 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:23.870196 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:23.992018 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:23.991981 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:24.992323 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:24.992261 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-lkjhw" Sep 18 17:22:27.664407 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.664372 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2"] Sep 18 17:22:27.664908 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.664699 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" podUID="e3ff6443-cd2f-4900-b5b1-4be46b03dc91" containerName="manager" containerID="cri-o://6bfd1e9db2b96c9be829152d04b485ca740b4947fef909f1d5bd34c1684ed55b" gracePeriod=2 Sep 18 17:22:27.691713 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.691680 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2"] Sep 18 17:22:27.695380 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:22:27.695339 2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"e3ff6443-cd2f-4900-b5b1-4be46b03dc91\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-77zb2\": pods \"limitador-operator-controller-manager-74495c69f8-77zb2\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:27.759374 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.759340 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6blrb"] Sep 18 17:22:27.759754 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.759742 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ff6443-cd2f-4900-b5b1-4be46b03dc91" containerName="manager" Sep 18 17:22:27.759797 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.759755 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ff6443-cd2f-4900-b5b1-4be46b03dc91" containerName="manager" Sep 18 17:22:27.759852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.759843 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ff6443-cd2f-4900-b5b1-4be46b03dc91" containerName="manager" Sep 18 17:22:27.762920 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.762900 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6blrb" Sep 18 17:22:27.799803 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.799771 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6blrb"] Sep 18 17:22:27.847616 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.847570 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7l2v\" (UniqueName: \"kubernetes.io/projected/9ad315f6-9ff4-4a1a-a73d-32e66a02dd82-kube-api-access-v7l2v\") pod \"limitador-operator-controller-manager-74495c69f8-6blrb\" (UID: \"9ad315f6-9ff4-4a1a-a73d-32e66a02dd82\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6blrb" Sep 18 17:22:27.847801 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:22:27.847654 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-77zb2\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="e3ff6443-cd2f-4900-b5b1-4be46b03dc91" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:27.924816 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.924756 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:27.938009 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:22:27.937970 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-77zb2\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="e3ff6443-cd2f-4900-b5b1-4be46b03dc91" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:27.948401 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.948372 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8q67\" (UniqueName: \"kubernetes.io/projected/e3ff6443-cd2f-4900-b5b1-4be46b03dc91-kube-api-access-z8q67\") pod \"e3ff6443-cd2f-4900-b5b1-4be46b03dc91\" (UID: \"e3ff6443-cd2f-4900-b5b1-4be46b03dc91\") " Sep 18 17:22:27.948601 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.948586 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7l2v\" (UniqueName: \"kubernetes.io/projected/9ad315f6-9ff4-4a1a-a73d-32e66a02dd82-kube-api-access-v7l2v\") pod \"limitador-operator-controller-manager-74495c69f8-6blrb\" (UID: \"9ad315f6-9ff4-4a1a-a73d-32e66a02dd82\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6blrb" Sep 18 17:22:27.950574 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.950549 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ff6443-cd2f-4900-b5b1-4be46b03dc91-kube-api-access-z8q67" (OuterVolumeSpecName: "kube-api-access-z8q67") pod "e3ff6443-cd2f-4900-b5b1-4be46b03dc91" (UID: "e3ff6443-cd2f-4900-b5b1-4be46b03dc91"). InnerVolumeSpecName "kube-api-access-z8q67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:22:27.978899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:27.978865 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7l2v\" (UniqueName: \"kubernetes.io/projected/9ad315f6-9ff4-4a1a-a73d-32e66a02dd82-kube-api-access-v7l2v\") pod \"limitador-operator-controller-manager-74495c69f8-6blrb\" (UID: \"9ad315f6-9ff4-4a1a-a73d-32e66a02dd82\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6blrb" Sep 18 17:22:28.049693 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:28.049663 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8q67\" (UniqueName: \"kubernetes.io/projected/e3ff6443-cd2f-4900-b5b1-4be46b03dc91-kube-api-access-z8q67\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:22:28.059115 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:28.059079 2560 generic.go:356] "Generic (PLEG): container finished" podID="e3ff6443-cd2f-4900-b5b1-4be46b03dc91" containerID="6bfd1e9db2b96c9be829152d04b485ca740b4947fef909f1d5bd34c1684ed55b" exitCode=0 Sep 18 17:22:28.059285 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:28.059125 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:28.059285 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:28.059177 2560 scope.go:117] "RemoveContainer" containerID="6bfd1e9db2b96c9be829152d04b485ca740b4947fef909f1d5bd34c1684ed55b" Sep 18 17:22:28.064425 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:22:28.064395 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-77zb2\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="e3ff6443-cd2f-4900-b5b1-4be46b03dc91" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:28.068421 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:28.068403 2560 scope.go:117] "RemoveContainer" containerID="6bfd1e9db2b96c9be829152d04b485ca740b4947fef909f1d5bd34c1684ed55b" Sep 18 17:22:28.068720 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:22:28.068701 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfd1e9db2b96c9be829152d04b485ca740b4947fef909f1d5bd34c1684ed55b\": container with ID starting with 6bfd1e9db2b96c9be829152d04b485ca740b4947fef909f1d5bd34c1684ed55b not found: ID does not exist" containerID="6bfd1e9db2b96c9be829152d04b485ca740b4947fef909f1d5bd34c1684ed55b" Sep 18 17:22:28.068804 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:28.068730 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfd1e9db2b96c9be829152d04b485ca740b4947fef909f1d5bd34c1684ed55b"} err="failed to get container status \"6bfd1e9db2b96c9be829152d04b485ca740b4947fef909f1d5bd34c1684ed55b\": rpc error: code = NotFound desc = could not find container \"6bfd1e9db2b96c9be829152d04b485ca740b4947fef909f1d5bd34c1684ed55b\": container with ID starting with 6bfd1e9db2b96c9be829152d04b485ca740b4947fef909f1d5bd34c1684ed55b not found: ID does not exist" Sep 18 17:22:28.070734 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:22:28.070713 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-77zb2\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="e3ff6443-cd2f-4900-b5b1-4be46b03dc91" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:28.073860 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:28.073835 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6blrb" Sep 18 17:22:28.387835 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:28.387808 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6blrb"] Sep 18 17:22:28.389707 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:22:28.389685 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad315f6_9ff4_4a1a_a73d_32e66a02dd82.slice/crio-698d8f0cb3912a4581e57cc7e119ff55f1f0b10d19f12acc2fd4cad840e61853 WatchSource:0}: Error finding container 698d8f0cb3912a4581e57cc7e119ff55f1f0b10d19f12acc2fd4cad840e61853: Status 404 returned error can't find the container with id 698d8f0cb3912a4581e57cc7e119ff55f1f0b10d19f12acc2fd4cad840e61853 Sep 18 17:22:28.864311 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:28.864267 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3ff6443-cd2f-4900-b5b1-4be46b03dc91" path="/var/lib/kubelet/pods/e3ff6443-cd2f-4900-b5b1-4be46b03dc91/volumes" Sep 18 17:22:28.864856 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:22:28.864835 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-77zb2\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="e3ff6443-cd2f-4900-b5b1-4be46b03dc91" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-77zb2" Sep 18 17:22:29.067334 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:29.067275 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6blrb" event={"ID":"9ad315f6-9ff4-4a1a-a73d-32e66a02dd82","Type":"ContainerStarted","Data":"0a8d62fdc932f8219daadec24ecdb44cc3ea05cb92c62d16fb895dda4a12f9c8"} Sep 18 17:22:29.067496 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:29.067349 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6blrb" event={"ID":"9ad315f6-9ff4-4a1a-a73d-32e66a02dd82","Type":"ContainerStarted","Data":"698d8f0cb3912a4581e57cc7e119ff55f1f0b10d19f12acc2fd4cad840e61853"} Sep 18 17:22:29.067496 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:29.067459 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6blrb" Sep 18 17:22:29.123702 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:29.123542 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6blrb" podStartSLOduration=2.12352369 podStartE2EDuration="2.12352369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:22:29.121352543 +0000 UTC m=+462.374513381" watchObservedRunningTime="2026-09-18 17:22:29.12352369 +0000 UTC m=+462.376684506" Sep 18 17:22:30.836189 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:30.836161 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:30.997144 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:30.997114 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:32.331578 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:32.331545 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9"] Sep 18 17:22:32.335166 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:32.335147 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:22:32.344986 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:32.344968 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v5c59\"" Sep 18 17:22:32.353675 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:32.353651 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9"] Sep 18 17:22:32.386664 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:32.386623 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e88128b-b901-4a7d-905d-53641faa452b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-md8r9\" (UID: \"7e88128b-b901-4a7d-905d-53641faa452b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:22:32.386664 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:32.386661 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xzk\" (UniqueName: \"kubernetes.io/projected/7e88128b-b901-4a7d-905d-53641faa452b-kube-api-access-29xzk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-md8r9\" (UID: \"7e88128b-b901-4a7d-905d-53641faa452b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:22:32.487691 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:32.487655 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e88128b-b901-4a7d-905d-53641faa452b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-md8r9\" (UID: \"7e88128b-b901-4a7d-905d-53641faa452b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:22:32.487691 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:32.487693 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xzk\" (UniqueName: \"kubernetes.io/projected/7e88128b-b901-4a7d-905d-53641faa452b-kube-api-access-29xzk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-md8r9\" (UID: \"7e88128b-b901-4a7d-905d-53641faa452b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:22:32.488051 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:32.488028 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e88128b-b901-4a7d-905d-53641faa452b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-md8r9\" (UID: \"7e88128b-b901-4a7d-905d-53641faa452b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:22:32.520410 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:32.520375 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xzk\" (UniqueName: \"kubernetes.io/projected/7e88128b-b901-4a7d-905d-53641faa452b-kube-api-access-29xzk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-md8r9\" (UID: \"7e88128b-b901-4a7d-905d-53641faa452b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:22:32.645346 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:32.645222 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:22:33.006177 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:33.006139 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9"] Sep 18 17:22:33.010236 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:22:33.010207 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e88128b_b901_4a7d_905d_53641faa452b.slice/crio-1108b7e7549cf015bc49ab69e5596c409afeec200de23aaa050b38c7bcc055b2 WatchSource:0}: Error finding container 1108b7e7549cf015bc49ab69e5596c409afeec200de23aaa050b38c7bcc055b2: Status 404 returned error can't find the container with id 1108b7e7549cf015bc49ab69e5596c409afeec200de23aaa050b38c7bcc055b2 Sep 18 17:22:33.090127 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:33.090094 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" event={"ID":"7e88128b-b901-4a7d-905d-53641faa452b","Type":"ContainerStarted","Data":"1108b7e7549cf015bc49ab69e5596c409afeec200de23aaa050b38c7bcc055b2"} Sep 18 17:22:35.589339 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:35.589307 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:35.685532 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:35.685500 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:35.958840 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:35.958807 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:36.077447 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:36.077415 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:38.119358 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:38.119248 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" event={"ID":"7e88128b-b901-4a7d-905d-53641faa452b","Type":"ContainerStarted","Data":"347ca721606971b4f16d7fd727f0a748e2f20c9fe0a942ed09a1e8a11920d682"} Sep 18 17:22:38.119358 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:38.119337 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:22:38.152597 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:38.152537 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" podStartSLOduration=1.477800458 podStartE2EDuration="6.152521185s" totalImagesPullingTime="4.674720727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:22:38.149887682 +0000 UTC m=+471.403048499" watchObservedRunningTime="2026-09-18 17:22:38.152521185 +0000 UTC m=+471.405682003" Sep 18 17:22:40.005736 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:40.005708 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:40.074629 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:40.074601 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6blrb" Sep 18 17:22:40.309370 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:40.309280 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:45.596379 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:45.596345 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:45.693472 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:45.693441 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:45.985850 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:45.985821 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:46.080619 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:46.080589 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:46.378326 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:46.378226 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:46.481256 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:46.481223 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:49.127213 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:49.127133 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:22:52.247810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:52.247776 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:52.348900 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:52.348871 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:57.231848 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:57.231814 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:22:57.332904 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:22:57.332873 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:00.911799 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:00.911771 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:01.012645 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:01.012617 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:05.770478 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:05.770448 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:05.917417 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:05.917381 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:08.969714 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:08.969685 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:09.115814 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.115786 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:09.225554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.225468 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:09.344183 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.344143 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:23:09.347249 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.347226 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvxk8" Sep 18 17:23:09.350330 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.350287 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-flxjs\"" Sep 18 17:23:09.350497 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.350477 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 18 17:23:09.358225 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.358199 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:23:09.420347 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.420316 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:23:09.440185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.440152 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d49821d9-884c-4e55-bea6-ca95b943e2e5-config-file\") pod \"limitador-limitador-84994f7ffc-rvxk8\" (UID: \"d49821d9-884c-4e55-bea6-ca95b943e2e5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rvxk8" Sep 18 17:23:09.440372 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.440213 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdqmp\" (UniqueName: \"kubernetes.io/projected/d49821d9-884c-4e55-bea6-ca95b943e2e5-kube-api-access-sdqmp\") pod \"limitador-limitador-84994f7ffc-rvxk8\" (UID: \"d49821d9-884c-4e55-bea6-ca95b943e2e5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rvxk8" Sep 18 17:23:09.540858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.540778 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdqmp\" (UniqueName: \"kubernetes.io/projected/d49821d9-884c-4e55-bea6-ca95b943e2e5-kube-api-access-sdqmp\") pod \"limitador-limitador-84994f7ffc-rvxk8\" (UID: \"d49821d9-884c-4e55-bea6-ca95b943e2e5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rvxk8" Sep 18 17:23:09.541008 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.540888 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d49821d9-884c-4e55-bea6-ca95b943e2e5-config-file\") pod \"limitador-limitador-84994f7ffc-rvxk8\" (UID: \"d49821d9-884c-4e55-bea6-ca95b943e2e5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rvxk8" Sep 18 17:23:09.541497 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.541477 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d49821d9-884c-4e55-bea6-ca95b943e2e5-config-file\") pod \"limitador-limitador-84994f7ffc-rvxk8\" (UID: \"d49821d9-884c-4e55-bea6-ca95b943e2e5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rvxk8" Sep 18 17:23:09.549671 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.549648 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdqmp\" (UniqueName: \"kubernetes.io/projected/d49821d9-884c-4e55-bea6-ca95b943e2e5-kube-api-access-sdqmp\") pod \"limitador-limitador-84994f7ffc-rvxk8\" (UID: \"d49821d9-884c-4e55-bea6-ca95b943e2e5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-rvxk8" Sep 18 17:23:09.658030 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.657990 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvxk8" Sep 18 17:23:09.885874 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.885781 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:09.929676 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.929646 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:23:09.931762 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:23:09.931731 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49821d9_884c_4e55_bea6_ca95b943e2e5.slice/crio-f7f841245dea7029ceeae222c7c5e8e0530f775508850892dbdcc350edc9ae6f WatchSource:0}: Error finding container f7f841245dea7029ceeae222c7c5e8e0530f775508850892dbdcc350edc9ae6f: Status 404 returned error can't find the container with id f7f841245dea7029ceeae222c7c5e8e0530f775508850892dbdcc350edc9ae6f Sep 18 17:23:09.962261 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:09.962228 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:10.282778 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:10.282744 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvxk8" event={"ID":"d49821d9-884c-4e55-bea6-ca95b943e2e5","Type":"ContainerStarted","Data":"f7f841245dea7029ceeae222c7c5e8e0530f775508850892dbdcc350edc9ae6f"} Sep 18 17:23:11.289350 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:11.289318 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvxk8" event={"ID":"d49821d9-884c-4e55-bea6-ca95b943e2e5","Type":"ContainerStarted","Data":"302666c920e387a5049e1a9669e2c2dae8fdd83bead72e8f349ad6b06268cef5"} Sep 18 17:23:11.329010 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:11.328955 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvxk8" podStartSLOduration=1.1280048759999999 podStartE2EDuration="2.328940024s" totalImagesPullingTime="1.200935148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:23:11.327474172 +0000 UTC m=+504.580635007" watchObservedRunningTime="2026-09-18 17:23:11.328940024 +0000 UTC m=+504.582100888" Sep 18 17:23:12.294192 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:12.294161 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvxk8" Sep 18 17:23:23.226972 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:23.226938 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:23.300389 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:23.300357 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-rvxk8" Sep 18 17:23:23.341003 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:23.340967 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:28.993414 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:28.993372 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:29.092858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:29.092826 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:29.453445 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:29.453406 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:42.283489 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:42.283453 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:42.629115 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:42.629031 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:48.978203 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:48.978132 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:49.071909 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:49.071868 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:49.173510 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:49.173455 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:49.242233 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:49.242151 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:49.309716 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:49.309680 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:52.753201 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:52.753173 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:52.848569 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:52.848493 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:23:57.321432 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:23:57.321392 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:22.651446 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:22.651412 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:22.740117 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:22.740088 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:24.229512 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:24.229477 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:24.322077 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:24.322043 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:24.533034 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:24.532951 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:24.664965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:24.664936 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:24.793321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:24.793237 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:24.915115 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:24.915087 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:24.959095 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:24.959063 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:24:29.222951 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:29.222917 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:35.258900 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:35.258867 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:35.340076 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:35.340046 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:45.194117 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:45.194087 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:45.776357 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:45.776325 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:24:48.819040 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:48.819012 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:24:48.819466 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:48.819015 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:24:48.825357 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:48.825339 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:24:48.825490 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:48.825401 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:24:54.162516 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:54.162484 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:24:54.548772 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:24:54.548671 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:25:00.032782 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.032701 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:25:00.367978 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.367894 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7"] Sep 18 17:25:00.371605 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.371583 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.377789 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.377767 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 18 17:25:00.378581 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.378559 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 18 17:25:00.378695 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.378618 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 18 17:25:00.378695 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.378592 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-htmgb\"" Sep 18 17:25:00.403685 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.403657 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7"] Sep 18 17:25:00.418362 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.418282 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:25:00.479524 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.479488 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2126dbfa-e868-42d2-8a36-0095b5de94fc-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.479704 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.479557 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2126dbfa-e868-42d2-8a36-0095b5de94fc-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.479704 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.479574 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd7tw\" (UniqueName: \"kubernetes.io/projected/2126dbfa-e868-42d2-8a36-0095b5de94fc-kube-api-access-wd7tw\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.479704 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.479592 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2126dbfa-e868-42d2-8a36-0095b5de94fc-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.479704 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.479631 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2126dbfa-e868-42d2-8a36-0095b5de94fc-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.479704 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.479649 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2126dbfa-e868-42d2-8a36-0095b5de94fc-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.580889 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.580848 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2126dbfa-e868-42d2-8a36-0095b5de94fc-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.580889 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.580894 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd7tw\" (UniqueName: \"kubernetes.io/projected/2126dbfa-e868-42d2-8a36-0095b5de94fc-kube-api-access-wd7tw\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.581161 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.581034 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2126dbfa-e868-42d2-8a36-0095b5de94fc-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.581161 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.581138 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2126dbfa-e868-42d2-8a36-0095b5de94fc-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.581265 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.581171 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2126dbfa-e868-42d2-8a36-0095b5de94fc-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.581265 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.581242 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2126dbfa-e868-42d2-8a36-0095b5de94fc-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.581617 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.581593 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2126dbfa-e868-42d2-8a36-0095b5de94fc-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.581699 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.581616 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2126dbfa-e868-42d2-8a36-0095b5de94fc-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.581741 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.581702 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2126dbfa-e868-42d2-8a36-0095b5de94fc-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.583849 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.583824 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2126dbfa-e868-42d2-8a36-0095b5de94fc-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.584050 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.584032 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2126dbfa-e868-42d2-8a36-0095b5de94fc-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.598368 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.598347 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd7tw\" (UniqueName: \"kubernetes.io/projected/2126dbfa-e868-42d2-8a36-0095b5de94fc-kube-api-access-wd7tw\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7\" (UID: \"2126dbfa-e868-42d2-8a36-0095b5de94fc\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.683441 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.683410 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:00.971575 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:00.971549 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7"] Sep 18 17:25:00.972283 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:25:00.972252 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2126dbfa_e868_42d2_8a36_0095b5de94fc.slice/crio-e7ce5371a24b8d66607829dc5ca7f88440ad018c187d052e835260851f94a4f1 WatchSource:0}: Error finding container e7ce5371a24b8d66607829dc5ca7f88440ad018c187d052e835260851f94a4f1: Status 404 returned error can't find the container with id e7ce5371a24b8d66607829dc5ca7f88440ad018c187d052e835260851f94a4f1 Sep 18 17:25:01.507532 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:01.507498 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:25:01.863684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:01.863598 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" event={"ID":"2126dbfa-e868-42d2-8a36-0095b5de94fc","Type":"ContainerStarted","Data":"e7ce5371a24b8d66607829dc5ca7f88440ad018c187d052e835260851f94a4f1"} Sep 18 17:25:02.869247 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:02.869207 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" event={"ID":"2126dbfa-e868-42d2-8a36-0095b5de94fc","Type":"ContainerStarted","Data":"9232109d9715bbda075a51db7ee297c7234f383e3fa97adf58db45d6546da93e"} Sep 18 17:25:02.898469 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:02.898404 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" podStartSLOduration=1.798092609 podStartE2EDuration="2.898385205s" totalImagesPullingTime="1.100292596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:25:02.89564871 +0000 UTC m=+616.148809528" watchObservedRunningTime="2026-09-18 17:25:02.898385205 +0000 UTC m=+616.151546023" Sep 18 17:25:05.118624 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:05.118588 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:25:06.434093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:06.434058 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:25:10.162077 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:10.162020 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:25:10.683535 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:10.683501 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:10.683702 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:10.683560 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:10.696026 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:10.695997 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:10.922520 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:10.922492 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7" Sep 18 17:25:11.904998 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:11.904951 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:25:15.121382 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:15.121343 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:25:20.049654 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:20.049609 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:25:22.010473 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:22.010444 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:25:30.881984 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:30.881953 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:25:33.277918 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:25:33.277878 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:26:24.357330 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:24.357273 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:24.456810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:24.456773 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:25.490764 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:25.490731 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:25.593449 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:25.593415 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:44.122080 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.122005 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:44.249054 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.249018 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:44.343904 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.343871 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:44.451615 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.451580 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7"] Sep 18 17:26:44.455123 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.455107 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.458034 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.458015 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-5b6d69-openshift-default-dockercfg-x2c7d\"" Sep 18 17:26:44.470833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.470809 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84m6b\" (UniqueName: \"kubernetes.io/projected/ad8c0339-0a9d-462f-922e-2359d1ead55b-kube-api-access-84m6b\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.470934 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.470858 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-credential-socket\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.470984 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.470927 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-workload-socket\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.470984 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.470951 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-token\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.470984 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.470977 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-data\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.471123 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.471004 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-envoy\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.471123 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.471025 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-workload-certs\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.471240 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.471115 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-podinfo\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.471240 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.471172 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8c0339-0a9d-462f-922e-2359d1ead55b-istiod-ca-cert\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.474731 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.474709 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7"] Sep 18 17:26:44.494479 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.494442 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:44.559906 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.559876 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj"] Sep 18 17:26:44.563414 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.563394 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.568629 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.568611 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-636b80-openshift-default-dockercfg-pdd28\"" Sep 18 17:26:44.572364 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572343 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84m6b\" (UniqueName: \"kubernetes.io/projected/ad8c0339-0a9d-462f-922e-2359d1ead55b-kube-api-access-84m6b\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.572447 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572379 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-token\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.572515 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572499 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nslr2\" (UniqueName: \"kubernetes.io/projected/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-kube-api-access-nslr2\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.572549 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572533 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-credential-socket\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.572587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572567 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-workload-socket\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.572587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572584 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-token\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.572678 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572608 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-workload-certs\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.572678 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572627 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-data\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.572678 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572650 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-workload-socket\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.572795 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572750 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-envoy\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.572795 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572775 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-data\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.572871 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572798 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-workload-certs\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.572922 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572903 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-credential-socket\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.572978 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.572955 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-data\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.573122 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.573102 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-workload-socket\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.573213 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.573148 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-workload-certs\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.573213 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.573151 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-envoy\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.573213 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.573190 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istiod-ca-cert\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.573555 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.573227 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-podinfo\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.573555 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.573275 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-podinfo\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.573555 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.573333 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8c0339-0a9d-462f-922e-2359d1ead55b-istiod-ca-cert\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.573555 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.573356 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-credential-socket\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.574105 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.574084 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8c0339-0a9d-462f-922e-2359d1ead55b-istiod-ca-cert\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.575170 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.575148 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-envoy\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.575342 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.575325 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-podinfo\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.587623 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.587601 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj"] Sep 18 17:26:44.591376 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.591355 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-token\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.591552 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.591536 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84m6b\" (UniqueName: \"kubernetes.io/projected/ad8c0339-0a9d-462f-922e-2359d1ead55b-kube-api-access-84m6b\") pod \"e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.601753 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.601727 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:44.674859 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.674822 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-workload-certs\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.675021 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.674866 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-workload-socket\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.675021 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.674891 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-data\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.675021 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.674929 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-envoy\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.675021 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.674946 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istiod-ca-cert\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.675021 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.674985 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-podinfo\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.675367 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.675025 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-credential-socket\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.675367 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.675057 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-token\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.675367 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.675086 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nslr2\" (UniqueName: \"kubernetes.io/projected/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-kube-api-access-nslr2\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.675367 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.675230 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-workload-certs\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.675367 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.675333 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-workload-socket\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.675367 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.675337 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-data\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.675686 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.675530 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-credential-socket\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.675844 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.675825 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istiod-ca-cert\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.677189 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.677171 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-envoy\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.677646 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.677630 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-podinfo\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.688839 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.688816 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:44.690411 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.690392 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-token\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.690632 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.690614 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nslr2\" (UniqueName: \"kubernetes.io/projected/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-kube-api-access-nslr2\") pod \"e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.765881 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.765818 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:44.788664 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.788641 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:44.873890 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.873859 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:44.888522 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:44.888496 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:45.048289 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.048264 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7"] Sep 18 17:26:45.050554 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:26:45.050530 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8c0339_0a9d_462f_922e_2359d1ead55b.slice/crio-3e78e64c4e8fe611ce079b7e7d2a91a1393b43e9c88fd053271281b16fc0eadd WatchSource:0}: Error finding container 3e78e64c4e8fe611ce079b7e7d2a91a1393b43e9c88fd053271281b16fc0eadd: Status 404 returned error can't find the container with id 3e78e64c4e8fe611ce079b7e7d2a91a1393b43e9c88fd053271281b16fc0eadd Sep 18 17:26:45.052923 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.052886 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:26:45.052980 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.052968 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:26:45.053035 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.053016 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:26:45.158909 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.158874 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj"] Sep 18 17:26:45.160089 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:26:45.160060 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088b4a2f_0f19_4224_a9b8_4108e29dcd8e.slice/crio-1613a0d10b6ea056e6fd43362a43d57ce826951da176c25e9b867576b2bd7077 WatchSource:0}: Error finding container 1613a0d10b6ea056e6fd43362a43d57ce826951da176c25e9b867576b2bd7077: Status 404 returned error can't find the container with id 1613a0d10b6ea056e6fd43362a43d57ce826951da176c25e9b867576b2bd7077 Sep 18 17:26:45.162798 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.162756 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:26:45.162871 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.162835 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:26:45.162914 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.162876 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:26:45.408791 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.408762 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" event={"ID":"ad8c0339-0a9d-462f-922e-2359d1ead55b","Type":"ContainerStarted","Data":"535261ffb1094ba1a880bb0348595c251a2ccd8a239c54cd32b162ed8ade2cce"} Sep 18 17:26:45.408982 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.408821 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" event={"ID":"ad8c0339-0a9d-462f-922e-2359d1ead55b","Type":"ContainerStarted","Data":"3e78e64c4e8fe611ce079b7e7d2a91a1393b43e9c88fd053271281b16fc0eadd"} Sep 18 17:26:45.411618 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.411569 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:45.411805 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.411777 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" event={"ID":"088b4a2f-0f19-4224-a9b8-4108e29dcd8e","Type":"ContainerStarted","Data":"961654a2e9944787031aa0890ce1871040030c17c0e272dfe0faf25bed5349f3"} Sep 18 17:26:45.411953 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.411830 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" event={"ID":"088b4a2f-0f19-4224-a9b8-4108e29dcd8e","Type":"ContainerStarted","Data":"1613a0d10b6ea056e6fd43362a43d57ce826951da176c25e9b867576b2bd7077"} Sep 18 17:26:45.469700 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.469650 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" podStartSLOduration=1.469631837 podStartE2EDuration="1.469631837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:45.468358925 +0000 UTC m=+718.721519763" watchObservedRunningTime="2026-09-18 17:26:45.469631837 +0000 UTC m=+718.722792655" Sep 18 17:26:45.470447 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.470403 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" podStartSLOduration=1.4703901670000001 podStartE2EDuration="1.470390167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:45.444916404 +0000 UTC m=+718.698077246" watchObservedRunningTime="2026-09-18 17:26:45.470390167 +0000 UTC m=+718.723550988" Sep 18 17:26:45.514815 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.514783 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:45.610243 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.610194 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:45.766206 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.766132 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:45.874340 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:45.874315 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:46.013252 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:46.013222 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:46.771553 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:46.771522 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:46.879253 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:46.879221 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:47.423324 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:47.423268 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:47.423500 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:47.423348 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:47.424032 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:47.424011 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:26:47.424120 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:47.424067 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:26:47.630383 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:47.630348 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:47.726946 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:47.726849 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:47.967982 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:47.967950 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:48.064153 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:48.064075 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:48.155907 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:48.155875 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:26:48.160446 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:48.160420 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:48.934745 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:48.934713 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:49.042310 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:49.042257 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:49.139404 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:49.139368 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:49.579829 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:49.579793 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:49.674648 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:49.674617 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:50.126606 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.126555 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2"] Sep 18 17:26:50.130390 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.130372 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.133782 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.133762 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-636b80-serving-cert\"" Sep 18 17:26:50.134984 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.134968 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 17:26:50.134984 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.134981 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 18 17:26:50.135129 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.134975 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sjtq4\"" Sep 18 17:26:50.135174 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.135146 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 18 17:26:50.143916 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.143894 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2"] Sep 18 17:26:50.225201 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.225170 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:50.234642 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.234607 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-maas-api-tls\") pod \"maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.234829 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.234811 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5zsr\" (UniqueName: \"kubernetes.io/projected/33fedc34-3591-43fb-a673-66fcdfe3a18c-kube-api-access-w5zsr\") pod \"maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.234966 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.234937 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fedc34-3591-43fb-a673-66fcdfe3a18c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.235029 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.235016 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-metrics-certs\") pod \"maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.306781 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.306751 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:50.335820 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.335782 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zsr\" (UniqueName: \"kubernetes.io/projected/33fedc34-3591-43fb-a673-66fcdfe3a18c-kube-api-access-w5zsr\") pod \"maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.336002 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.335829 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fedc34-3591-43fb-a673-66fcdfe3a18c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.336002 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.335869 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-metrics-certs\") pod \"maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.336002 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.335950 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-maas-api-tls\") pod \"maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.336195 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:26:50.336142 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-serving-cert: secret "maas-api-e2e-inf-a-636b80-serving-cert" not found Sep 18 17:26:50.336347 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:26:50.336260 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-maas-api-tls podName:33fedc34-3591-43fb-a673-66fcdfe3a18c nodeName:}" failed. No retries permitted until 2026-09-18 17:26:50.836238735 +0000 UTC m=+724.089399550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-maas-api-tls") pod "maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" (UID: "33fedc34-3591-43fb-a673-66fcdfe3a18c") : secret "maas-api-e2e-inf-a-636b80-serving-cert" not found Sep 18 17:26:50.336592 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.336572 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fedc34-3591-43fb-a673-66fcdfe3a18c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.338547 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.338528 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-metrics-certs\") pod \"maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.345730 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.345708 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5zsr\" (UniqueName: \"kubernetes.io/projected/33fedc34-3591-43fb-a673-66fcdfe3a18c-kube-api-access-w5zsr\") pod \"maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.489465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.489426 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:50.841885 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.841795 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-maas-api-tls\") pod \"maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.844466 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.844439 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-maas-api-tls\") pod \"maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:50.864327 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.864279 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:50.969527 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:50.969495 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:51.040716 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:51.040682 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:51.358849 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:51.358826 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2"] Sep 18 17:26:51.361242 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:26:51.361210 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fedc34_3591_43fb_a673_66fcdfe3a18c.slice/crio-a10b22067888815d5da6e8b51ccb90a688b78e97f81a73e0e5c2b6b962b062af WatchSource:0}: Error finding container a10b22067888815d5da6e8b51ccb90a688b78e97f81a73e0e5c2b6b962b062af: Status 404 returned error can't find the container with id a10b22067888815d5da6e8b51ccb90a688b78e97f81a73e0e5c2b6b962b062af Sep 18 17:26:51.362832 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:51.362811 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 17:26:51.447131 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:51.447095 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" event={"ID":"33fedc34-3591-43fb-a673-66fcdfe3a18c","Type":"ContainerStarted","Data":"a10b22067888815d5da6e8b51ccb90a688b78e97f81a73e0e5c2b6b962b062af"} Sep 18 17:26:51.487344 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:51.487306 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:51.567893 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:51.567857 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:52.027285 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.027250 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:52.145464 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.145428 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:52.351881 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.351775 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q"] Sep 18 17:26:52.356460 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.356435 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" Sep 18 17:26:52.361887 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.361782 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-0a2e13-dockercfg-7wj8w\"" Sep 18 17:26:52.361887 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.361842 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-0a2e13\"" Sep 18 17:26:52.374723 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.374611 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q"] Sep 18 17:26:52.465196 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.465013 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mclht\" (UniqueName: \"kubernetes.io/projected/8e3b4dba-f46c-443d-9728-3a3fb1afe129-kube-api-access-mclht\") pod \"payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q\" (UID: \"8e3b4dba-f46c-443d-9728-3a3fb1afe129\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" Sep 18 17:26:52.465196 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.465128 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3b4dba-f46c-443d-9728-3a3fb1afe129-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q\" (UID: \"8e3b4dba-f46c-443d-9728-3a3fb1afe129\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" Sep 18 17:26:52.566721 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.566681 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mclht\" (UniqueName: \"kubernetes.io/projected/8e3b4dba-f46c-443d-9728-3a3fb1afe129-kube-api-access-mclht\") pod \"payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q\" (UID: \"8e3b4dba-f46c-443d-9728-3a3fb1afe129\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" Sep 18 17:26:52.566924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.566773 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3b4dba-f46c-443d-9728-3a3fb1afe129-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q\" (UID: \"8e3b4dba-f46c-443d-9728-3a3fb1afe129\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" Sep 18 17:26:52.567599 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.567573 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3b4dba-f46c-443d-9728-3a3fb1afe129-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q\" (UID: \"8e3b4dba-f46c-443d-9728-3a3fb1afe129\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" Sep 18 17:26:52.576767 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.576734 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mclht\" (UniqueName: \"kubernetes.io/projected/8e3b4dba-f46c-443d-9728-3a3fb1afe129-kube-api-access-mclht\") pod \"payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q\" (UID: \"8e3b4dba-f46c-443d-9728-3a3fb1afe129\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" Sep 18 17:26:52.673594 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.673559 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" Sep 18 17:26:52.842926 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:52.842894 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:53.285998 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.285964 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:53.563325 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.563214 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv"] Sep 18 17:26:53.569530 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.569499 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.573808 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.573615 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-535fd3-openshift-default-dockercfg-z7zrc\"" Sep 18 17:26:53.578989 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.578563 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv"] Sep 18 17:26:53.607448 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.607417 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:53.678834 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.678796 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-istio-envoy\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.679055 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.678848 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-istio-data\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.679055 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.678893 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-credential-socket\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.679055 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.679005 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57fd2f9-3bf7-4dee-8997-483002041249-istio-podinfo\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.679237 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.679076 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57fd2f9-3bf7-4dee-8997-483002041249-istiod-ca-cert\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.679237 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.679106 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-workload-certs\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.679237 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.679153 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmmz6\" (UniqueName: \"kubernetes.io/projected/c57fd2f9-3bf7-4dee-8997-483002041249-kube-api-access-nmmz6\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.679237 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.679228 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-workload-socket\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.679446 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.679274 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57fd2f9-3bf7-4dee-8997-483002041249-istio-token\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.780187 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.780150 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmmz6\" (UniqueName: \"kubernetes.io/projected/c57fd2f9-3bf7-4dee-8997-483002041249-kube-api-access-nmmz6\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.780402 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.780207 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-workload-socket\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.780402 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.780246 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57fd2f9-3bf7-4dee-8997-483002041249-istio-token\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.780402 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.780334 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-istio-envoy\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.780402 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.780365 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-istio-data\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.780545 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.780407 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-credential-socket\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.780545 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.780461 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57fd2f9-3bf7-4dee-8997-483002041249-istio-podinfo\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.780545 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.780526 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57fd2f9-3bf7-4dee-8997-483002041249-istiod-ca-cert\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.780640 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.780559 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-workload-certs\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.780675 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.780648 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-workload-socket\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.780709 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.780689 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-istio-data\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.780901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.780880 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-workload-certs\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.780981 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.780940 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-credential-socket\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.781331 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.781283 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57fd2f9-3bf7-4dee-8997-483002041249-istiod-ca-cert\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.782836 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.782815 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-istio-envoy\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.783136 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.783120 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57fd2f9-3bf7-4dee-8997-483002041249-istio-podinfo\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.817652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.817580 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:53.827194 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.827166 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57fd2f9-3bf7-4dee-8997-483002041249-istio-token\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.830375 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.830356 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmmz6\" (UniqueName: \"kubernetes.io/projected/c57fd2f9-3bf7-4dee-8997-483002041249-kube-api-access-nmmz6\") pod \"e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:53.887516 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:53.887443 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:54.112093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.112065 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:54.112326 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.112307 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q"] Sep 18 17:26:54.114027 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:26:54.114007 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3b4dba_f46c_443d_9728_3a3fb1afe129.slice/crio-1a7f371c2833a4134a8357e8f360519d5922494ec15633e93410cff7210fe088 WatchSource:0}: Error finding container 1a7f371c2833a4134a8357e8f360519d5922494ec15633e93410cff7210fe088: Status 404 returned error can't find the container with id 1a7f371c2833a4134a8357e8f360519d5922494ec15633e93410cff7210fe088 Sep 18 17:26:54.187102 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.187071 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv"] Sep 18 17:26:54.188275 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:26:54.188249 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57fd2f9_3bf7_4dee_8997_483002041249.slice/crio-b73a300ef932aca00d5880294406fffacedf60f45a8d4c4371901e191385e6aa WatchSource:0}: Error finding container b73a300ef932aca00d5880294406fffacedf60f45a8d4c4371901e191385e6aa: Status 404 returned error can't find the container with id b73a300ef932aca00d5880294406fffacedf60f45a8d4c4371901e191385e6aa Sep 18 17:26:54.190493 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.190461 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:26:54.190556 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.190527 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:26:54.190589 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.190576 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:26:54.474777 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.474736 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" event={"ID":"c57fd2f9-3bf7-4dee-8997-483002041249","Type":"ContainerStarted","Data":"7881323fd83b464db168f5a41b0718a2d0568e1a6528e67ed238d00169b01446"} Sep 18 17:26:54.474974 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.474808 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" event={"ID":"c57fd2f9-3bf7-4dee-8997-483002041249","Type":"ContainerStarted","Data":"b73a300ef932aca00d5880294406fffacedf60f45a8d4c4371901e191385e6aa"} Sep 18 17:26:54.477250 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.477224 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" event={"ID":"33fedc34-3591-43fb-a673-66fcdfe3a18c","Type":"ContainerStarted","Data":"cf2a6799c86c1e0caadda32b9ed2f681a81eb8a46f48dab89862f8de949c4522"} Sep 18 17:26:54.477476 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.477453 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:26:54.478517 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.478497 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" event={"ID":"8e3b4dba-f46c-443d-9728-3a3fb1afe129","Type":"ContainerStarted","Data":"1a7f371c2833a4134a8357e8f360519d5922494ec15633e93410cff7210fe088"} Sep 18 17:26:54.498267 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.498216 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" podStartSLOduration=1.498199029 podStartE2EDuration="1.498199029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:54.496489704 +0000 UTC m=+727.749650554" watchObservedRunningTime="2026-09-18 17:26:54.498199029 +0000 UTC m=+727.751359845" Sep 18 17:26:54.516930 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.516870 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" podStartSLOduration=1.997082741 podStartE2EDuration="4.516850404s" totalImagesPullingTime="2.519767663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:54.514373413 +0000 UTC m=+727.767534256" watchObservedRunningTime="2026-09-18 17:26:54.516850404 +0000 UTC m=+727.770011222" Sep 18 17:26:54.887985 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:54.887906 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:55.039795 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.039750 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:55.137495 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.137466 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:55.293595 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.293565 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:55.598817 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.598729 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v"] Sep 18 17:26:55.606244 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.606221 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.610277 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.610254 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5cfd4b5c-openshift-default-dockercfg-nr2d5\"" Sep 18 17:26:55.630480 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.630446 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v"] Sep 18 17:26:55.654249 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.654216 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:55.702932 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.702808 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-credential-socket\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.702932 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.702855 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgnzb\" (UniqueName: \"kubernetes.io/projected/ffb83f58-69fe-4648-a093-6fdd0239ba34-kube-api-access-jgnzb\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.702932 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.702901 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-envoy\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.703371 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.702977 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb83f58-69fe-4648-a093-6fdd0239ba34-istiod-ca-cert\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.703371 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.703005 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-workload-certs\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.703371 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.703073 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-workload-socket\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.703371 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.703139 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-token\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.703371 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.703241 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-data\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.705865 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.705797 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-podinfo\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.807039 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.806999 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-credential-socket\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.807219 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.807048 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgnzb\" (UniqueName: \"kubernetes.io/projected/ffb83f58-69fe-4648-a093-6fdd0239ba34-kube-api-access-jgnzb\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.807287 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.807244 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-envoy\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.807380 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.807320 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb83f58-69fe-4648-a093-6fdd0239ba34-istiod-ca-cert\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.807380 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.807351 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-workload-certs\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.807483 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.807396 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-workload-socket\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.807483 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.807437 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-token\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.807591 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.807490 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-credential-socket\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.807591 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.807576 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-data\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.807703 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.807665 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-podinfo\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.807761 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.807702 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-workload-certs\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.808089 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.808062 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-workload-socket\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.808201 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.808090 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb83f58-69fe-4648-a093-6fdd0239ba34-istiod-ca-cert\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.808337 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.808283 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-data\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.809960 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.809937 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-envoy\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.810399 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.810367 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-podinfo\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.820589 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.820551 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-token\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.822576 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.822544 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgnzb\" (UniqueName: \"kubernetes.io/projected/ffb83f58-69fe-4648-a093-6fdd0239ba34-kube-api-access-jgnzb\") pod \"e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.909377 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.909322 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:55.920794 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.920673 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:55.971191 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:55.971155 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:56.205589 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:56.205510 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:56.496740 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:56.496359 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:56.497380 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:56.497359 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:26:56.598078 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:56.598010 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:56.695328 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:56.695281 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:56.823873 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:56.823849 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v"] Sep 18 17:26:56.826060 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:26:56.826030 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb83f58_69fe_4648_a093_6fdd0239ba34.slice/crio-55b44608e6319a382bf883a9b9791ec60fac296f2e2f60d2bdb84d45a86d5131 WatchSource:0}: Error finding container 55b44608e6319a382bf883a9b9791ec60fac296f2e2f60d2bdb84d45a86d5131: Status 404 returned error can't find the container with id 55b44608e6319a382bf883a9b9791ec60fac296f2e2f60d2bdb84d45a86d5131 Sep 18 17:26:56.828827 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:56.828798 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:26:56.828897 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:56.828861 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:26:56.828897 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:56.828893 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:26:57.342554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:57.342520 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:57.503027 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:57.502989 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" event={"ID":"8e3b4dba-f46c-443d-9728-3a3fb1afe129","Type":"ContainerStarted","Data":"8190354c69605fbe6568ec16b7ddea98bd6b8279dc1ef96663267c824ccf3e55"} Sep 18 17:26:57.503232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:57.503058 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" Sep 18 17:26:57.504633 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:57.504601 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" event={"ID":"ffb83f58-69fe-4648-a093-6fdd0239ba34","Type":"ContainerStarted","Data":"02f29e9ab6b3901b37888aa096bedc317223cbdd2b47cf992b7b002d5ca9545e"} Sep 18 17:26:57.504765 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:57.504649 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" event={"ID":"ffb83f58-69fe-4648-a093-6fdd0239ba34","Type":"ContainerStarted","Data":"55b44608e6319a382bf883a9b9791ec60fac296f2e2f60d2bdb84d45a86d5131"} Sep 18 17:26:57.526940 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:57.526885 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" podStartSLOduration=3.048899569 podStartE2EDuration="5.52687001s" totalImagesPullingTime="2.477970441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:57.524907588 +0000 UTC m=+730.778068431" watchObservedRunningTime="2026-09-18 17:26:57.52687001 +0000 UTC m=+730.780030827" Sep 18 17:26:57.555349 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:57.551747 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" podStartSLOduration=2.551727007 podStartE2EDuration="2.551727007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:26:57.550863572 +0000 UTC m=+730.804024406" watchObservedRunningTime="2026-09-18 17:26:57.551727007 +0000 UTC m=+730.804887825" Sep 18 17:26:57.921566 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:57.921530 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:57.926689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:57.926661 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:58.456208 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:58.456171 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:58.509983 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:58.509952 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:58.510849 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:58.510830 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:26:58.989170 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:58.989121 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:59.091420 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:59.091375 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:26:59.852058 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:26:59.852016 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:00.499405 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:00.499375 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:27:01.081950 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:01.081851 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:01.167063 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:01.167025 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:01.599118 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:01.599070 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:01.709881 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:01.709793 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:01.796270 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:01.796202 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:02.117979 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:02.117942 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:02.193573 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:02.193520 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:02.295309 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:02.295261 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:02.475093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:02.475054 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:02.595406 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:02.595376 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:02.690881 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:02.690850 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:02.856370 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:02.856151 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:02.952194 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:02.952161 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:03.105993 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.105922 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:03.296788 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.296752 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:03.328560 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.328524 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7"] Sep 18 17:27:03.332791 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.332768 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.337429 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.337401 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2aa65a-openshift-default-dockercfg-h28c7\"" Sep 18 17:27:03.427929 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.427888 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-data\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.427929 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.427927 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-credential-socket\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.428185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.427951 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-workload-certs\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.428185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.428047 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istiod-ca-cert\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.428185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.428133 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-token\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.428185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.428163 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-workload-socket\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.428482 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.428198 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcb9n\" (UniqueName: \"kubernetes.io/projected/6d34c7af-3bbd-4d5a-bec3-b470994af0da-kube-api-access-vcb9n\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.428482 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.428249 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-envoy\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.428482 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.428334 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-podinfo\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.458859 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.458816 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:03.489001 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.488952 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7"] Sep 18 17:27:03.528901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.528867 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-podinfo\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.529098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.528923 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-data\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.529098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.528958 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-credential-socket\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.529098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.528998 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-workload-certs\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.529274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.529103 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istiod-ca-cert\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.529274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.529190 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-token\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.529274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.529221 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-workload-socket\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.529274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.529253 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcb9n\" (UniqueName: \"kubernetes.io/projected/6d34c7af-3bbd-4d5a-bec3-b470994af0da-kube-api-access-vcb9n\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.529514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.529332 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-envoy\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.529514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.529436 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-workload-certs\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.529514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.529471 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-data\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.529955 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.529859 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-credential-socket\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.530092 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.529960 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-workload-socket\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.530215 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.530192 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istiod-ca-cert\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.531573 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.531554 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-envoy\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.531898 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.531880 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-podinfo\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.563423 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.561417 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcb9n\" (UniqueName: \"kubernetes.io/projected/6d34c7af-3bbd-4d5a-bec3-b470994af0da-kube-api-access-vcb9n\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.564835 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.564759 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-token\") pod \"e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.629037 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.629004 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:03.659612 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.659568 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:03.839700 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.839578 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:03.967339 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:27:03.967274 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d34c7af_3bbd_4d5a_bec3_b470994af0da.slice/crio-cb115eda343c22b279e73144851924c48cdaec73a06273c02a245b0380170d40 WatchSource:0}: Error finding container cb115eda343c22b279e73144851924c48cdaec73a06273c02a245b0380170d40: Status 404 returned error can't find the container with id cb115eda343c22b279e73144851924c48cdaec73a06273c02a245b0380170d40 Sep 18 17:27:03.970478 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.970453 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7"] Sep 18 17:27:03.970661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.970491 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:03.970861 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.970836 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:03.971016 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:03.970998 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:04.076083 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:04.076047 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:04.556662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:04.556618 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" event={"ID":"6d34c7af-3bbd-4d5a-bec3-b470994af0da","Type":"ContainerStarted","Data":"54fae24d3592f8f51db53690294b32e2f2d4824cb3168653e1c61acceb458ca8"} Sep 18 17:27:04.557189 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:04.556697 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" event={"ID":"6d34c7af-3bbd-4d5a-bec3-b470994af0da","Type":"ContainerStarted","Data":"cb115eda343c22b279e73144851924c48cdaec73a06273c02a245b0380170d40"} Sep 18 17:27:04.606365 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:04.606313 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" podStartSLOduration=1.606280291 podStartE2EDuration="1.606280291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:04.605069453 +0000 UTC m=+737.858230273" watchObservedRunningTime="2026-09-18 17:27:04.606280291 +0000 UTC m=+737.859441108" Sep 18 17:27:04.660407 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:04.660375 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:05.403720 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:05.403688 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:05.667824 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:05.667792 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:06.342914 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:06.342876 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:06.569651 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:06.569590 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:06.571195 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:06.571167 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:06.698377 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:06.698332 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:06.800063 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:06.800033 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:07.254357 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:07.254320 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:27:07.580075 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:07.579991 2560 generic.go:356] "Generic (PLEG): container finished" podID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerID="961654a2e9944787031aa0890ce1871040030c17c0e272dfe0faf25bed5349f3" exitCode=0 Sep 18 17:27:07.580241 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:07.580067 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" event={"ID":"088b4a2f-0f19-4224-a9b8-4108e29dcd8e","Type":"ContainerDied","Data":"961654a2e9944787031aa0890ce1871040030c17c0e272dfe0faf25bed5349f3"} Sep 18 17:27:07.580708 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:07.580677 2560 scope.go:117] "RemoveContainer" containerID="961654a2e9944787031aa0890ce1871040030c17c0e272dfe0faf25bed5349f3" Sep 18 17:27:07.582082 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:07.582056 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:07.582142 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:07.582124 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:07.582190 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:07.582179 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:07.657423 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:07.654264 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:07.811029 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:07.811000 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:07.946934 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:07.946899 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:08.043061 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:08.043029 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:08.144674 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:08.144626 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:08.248264 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:08.248162 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:08.494080 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:08.494036 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:08.513124 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:08.513034 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" Sep 18 17:27:08.589862 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:08.589796 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:08.591679 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:08.591659 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" event={"ID":"088b4a2f-0f19-4224-a9b8-4108e29dcd8e","Type":"ContainerStarted","Data":"f54fbb96852e2587307c6c8b661324856744da47848aea371ad2e6be42b24574"} Sep 18 17:27:08.874801 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:08.874695 2560 patch_prober.go:28] interesting pod/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 18 17:27:08.875350 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:08.875318 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 18 17:27:08.875767 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:08.875187 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:27:08.980348 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:08.980311 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:09.606320 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:09.604513 2560 generic.go:356] "Generic (PLEG): container finished" podID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerID="535261ffb1094ba1a880bb0348595c251a2ccd8a239c54cd32b162ed8ade2cce" exitCode=0 Sep 18 17:27:09.606320 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:09.605370 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" event={"ID":"ad8c0339-0a9d-462f-922e-2359d1ead55b","Type":"ContainerDied","Data":"535261ffb1094ba1a880bb0348595c251a2ccd8a239c54cd32b162ed8ade2cce"} Sep 18 17:27:09.606320 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:09.605813 2560 scope.go:117] "RemoveContainer" containerID="535261ffb1094ba1a880bb0348595c251a2ccd8a239c54cd32b162ed8ade2cce" Sep 18 17:27:09.607960 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:09.607572 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:09.607960 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:09.607665 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:09.607960 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:09.607752 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:09.874760 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:09.874665 2560 patch_prober.go:28] interesting pod/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 18 17:27:09.874760 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:09.874734 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 18 17:27:10.159431 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:10.155923 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:10.617042 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:10.616927 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" event={"ID":"ad8c0339-0a9d-462f-922e-2359d1ead55b","Type":"ContainerStarted","Data":"77599988ecb09e76cdb39982ba71f98bd489e408ea60af4250fc6ccf4e5c37bb"} Sep 18 17:27:10.767312 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:10.767249 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:27:10.767532 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:10.767335 2560 patch_prober.go:28] interesting pod/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 18 17:27:10.767532 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:10.767397 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 18 17:27:10.870687 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:10.869513 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:10.875222 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:10.875184 2560 patch_prober.go:28] interesting pod/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 18 17:27:10.875830 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:10.875795 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 18 17:27:11.039057 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:11.039022 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:11.187857 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:11.187775 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:11.290618 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:11.290579 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:11.393246 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:11.393217 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:11.570465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:11.570347 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:11.767268 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:11.767227 2560 patch_prober.go:28] interesting pod/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 18 17:27:11.767572 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:11.767539 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 18 17:27:11.875287 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:11.875168 2560 patch_prober.go:28] interesting pod/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 18 17:27:11.875901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:11.875870 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 18 17:27:12.405328 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.402635 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq"] Sep 18 17:27:12.416314 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.415738 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:12.432404 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.427055 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2aa65a-serving-cert\"" Sep 18 17:27:12.457773 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.456197 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq"] Sep 18 17:27:12.514860 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.514826 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:12.565321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.561365 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-metrics-certs\") pod \"maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:12.565321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.561437 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033aacb7-d7be-4cb3-b990-235483f5ddeb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:12.565321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.561479 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6cw7\" (UniqueName: \"kubernetes.io/projected/033aacb7-d7be-4cb3-b990-235483f5ddeb-kube-api-access-x6cw7\") pod \"maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:12.565321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.561684 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-maas-api-tls\") pod \"maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:12.630903 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.630849 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:12.665006 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.664898 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-maas-api-tls\") pod \"maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:12.666141 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.666113 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-metrics-certs\") pod \"maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:12.666803 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.666784 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033aacb7-d7be-4cb3-b990-235483f5ddeb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:12.666962 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.666947 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6cw7\" (UniqueName: \"kubernetes.io/projected/033aacb7-d7be-4cb3-b990-235483f5ddeb-kube-api-access-x6cw7\") pod \"maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:12.667566 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:27:12.665336 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-serving-cert: secret "maas-api-e2e-inf-b-2aa65a-serving-cert" not found Sep 18 17:27:12.667744 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:27:12.667731 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-maas-api-tls podName:033aacb7-d7be-4cb3-b990-235483f5ddeb nodeName:}" failed. No retries permitted until 2026-09-18 17:27:13.167703847 +0000 UTC m=+746.420864647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-maas-api-tls") pod "maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" (UID: "033aacb7-d7be-4cb3-b990-235483f5ddeb") : secret "maas-api-e2e-inf-b-2aa65a-serving-cert" not found Sep 18 17:27:12.669251 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.668565 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033aacb7-d7be-4cb3-b990-235483f5ddeb-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:12.686316 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.683261 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6cw7\" (UniqueName: \"kubernetes.io/projected/033aacb7-d7be-4cb3-b990-235483f5ddeb-kube-api-access-x6cw7\") pod \"maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:12.692936 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.692895 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-metrics-certs\") pod \"maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:12.767117 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.767083 2560 patch_prober.go:28] interesting pod/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 18 17:27:12.767453 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.767419 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 18 17:27:12.875098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.875059 2560 patch_prober.go:28] interesting pod/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 18 17:27:12.875311 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:12.875123 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 18 17:27:13.032805 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:13.032670 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:13.175424 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:13.175379 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-maas-api-tls\") pod \"maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:13.179097 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:13.179060 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-maas-api-tls\") pod \"maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:13.362613 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:13.362509 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:13.785317 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:13.781422 2560 patch_prober.go:28] interesting pod/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 18 17:27:13.785317 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:13.781486 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 18 17:27:13.882962 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:13.882872 2560 patch_prober.go:28] interesting pod/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 18 17:27:13.883203 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:13.883172 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 18 17:27:13.935328 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:13.918875 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq"] Sep 18 17:27:13.935328 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:13.927850 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:14.034740 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:14.034610 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:14.604001 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:14.603733 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:14.764574 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:14.764475 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:14.767262 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:14.767227 2560 patch_prober.go:28] interesting pod/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 18 17:27:14.767430 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:14.767286 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 18 17:27:14.796951 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:14.796912 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" event={"ID":"033aacb7-d7be-4cb3-b990-235483f5ddeb","Type":"ContainerStarted","Data":"8345add3af5587ce05a4febfdacd486194f66872436b8409a6b16fca14e7a981"} Sep 18 17:27:14.797136 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:14.796980 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" event={"ID":"033aacb7-d7be-4cb3-b990-235483f5ddeb","Type":"ContainerStarted","Data":"690d503ce4da5ac152337c6e8af0a31ff672bf119cfb7b0f26f522eab3c6f4b0"} Sep 18 17:27:14.797316 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:14.797277 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:14.827127 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:14.827068 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" podStartSLOduration=2.328869842 podStartE2EDuration="2.827043616s" totalImagesPullingTime="498.173774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:14.823845415 +0000 UTC m=+748.077006251" watchObservedRunningTime="2026-09-18 17:27:14.827043616 +0000 UTC m=+748.080204527" Sep 18 17:27:15.084375 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:15.076931 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:27:15.084375 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:15.078276 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:27:15.096331 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:15.094561 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:27:15.700958 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:15.700916 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:15.766486 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:15.766447 2560 patch_prober.go:28] interesting pod/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 18 17:27:15.766683 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:15.766535 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 18 17:27:16.766806 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:16.766353 2560 patch_prober.go:28] interesting pod/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 18 17:27:16.766806 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:16.766433 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 18 17:27:16.774498 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:16.774402 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:17.423117 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:17.423081 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:27:17.767016 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:17.766930 2560 patch_prober.go:28] interesting pod/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 18 17:27:17.767016 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:17.767000 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 18 17:27:18.354884 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:18.354843 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:18.771418 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:18.771382 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:27:18.772483 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:18.772459 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:27:19.344131 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:19.344101 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:19.830943 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:19.830913 2560 generic.go:356] "Generic (PLEG): container finished" podID="c57fd2f9-3bf7-4dee-8997-483002041249" containerID="7881323fd83b464db168f5a41b0718a2d0568e1a6528e67ed238d00169b01446" exitCode=0 Sep 18 17:27:19.831389 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:19.830991 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" event={"ID":"c57fd2f9-3bf7-4dee-8997-483002041249","Type":"ContainerDied","Data":"7881323fd83b464db168f5a41b0718a2d0568e1a6528e67ed238d00169b01446"} Sep 18 17:27:19.831484 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:19.831465 2560 scope.go:117] "RemoveContainer" containerID="7881323fd83b464db168f5a41b0718a2d0568e1a6528e67ed238d00169b01446" Sep 18 17:27:19.832659 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:19.832634 2560 generic.go:356] "Generic (PLEG): container finished" podID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerID="02f29e9ab6b3901b37888aa096bedc317223cbdd2b47cf992b7b002d5ca9545e" exitCode=0 Sep 18 17:27:19.832766 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:19.832658 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" event={"ID":"ffb83f58-69fe-4648-a093-6fdd0239ba34","Type":"ContainerDied","Data":"02f29e9ab6b3901b37888aa096bedc317223cbdd2b47cf992b7b002d5ca9545e"} Sep 18 17:27:19.832766 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:19.832676 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:19.832766 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:19.832738 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:19.832869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:19.832770 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:19.833005 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:19.832982 2560 scope.go:117] "RemoveContainer" containerID="02f29e9ab6b3901b37888aa096bedc317223cbdd2b47cf992b7b002d5ca9545e" Sep 18 17:27:19.834087 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:19.834059 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:19.834185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:19.834113 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:19.834185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:19.834146 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:20.190481 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:20.190420 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:20.269620 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:20.269552 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:20.448795 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:20.448699 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:20.812224 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:20.812131 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:27:20.842958 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:20.842924 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" event={"ID":"c57fd2f9-3bf7-4dee-8997-483002041249","Type":"ContainerStarted","Data":"99eb3c25c9933dec8ef83d9d5007a1a94a38f4caf331160d6bfa04855b1c8005"} Sep 18 17:27:20.845916 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:20.845883 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" event={"ID":"ffb83f58-69fe-4648-a093-6fdd0239ba34","Type":"ContainerStarted","Data":"d22367a6c00d38cf81d7d3fa33c84c54e109f47dbd24653ae1ce206e4a9d6832"} Sep 18 17:27:20.890358 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:20.888157 2560 patch_prober.go:28] interesting pod/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 18 17:27:20.890358 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:20.888233 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 18 17:27:20.890358 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:20.888552 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:27:20.921723 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:20.921691 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:27:20.921905 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:20.921805 2560 patch_prober.go:28] interesting pod/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 18 17:27:20.921905 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:20.921846 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 18 17:27:21.595558 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:21.595515 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:21.889279 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:21.888758 2560 patch_prober.go:28] interesting pod/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 18 17:27:21.889279 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:21.888824 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 18 17:27:21.921536 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:21.921507 2560 patch_prober.go:28] interesting pod/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 18 17:27:21.921714 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:21.921561 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 18 17:27:22.040130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:22.040098 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:22.853582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:22.853498 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:22.888418 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:22.888381 2560 patch_prober.go:28] interesting pod/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 18 17:27:22.888689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:22.888661 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 18 17:27:22.921719 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:22.921679 2560 patch_prober.go:28] interesting pod/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 18 17:27:22.922084 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:22.921755 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 18 17:27:23.888900 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:23.888865 2560 patch_prober.go:28] interesting pod/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 18 17:27:23.889199 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:23.889171 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 18 17:27:23.921671 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:23.921640 2560 patch_prober.go:28] interesting pod/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 18 17:27:23.921930 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:23.921902 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 18 17:27:24.299925 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:24.299886 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:24.888876 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:24.888833 2560 patch_prober.go:28] interesting pod/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 18 17:27:24.889154 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:24.888902 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 18 17:27:24.921609 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:24.921573 2560 patch_prober.go:28] interesting pod/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 18 17:27:24.921933 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:24.921888 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 18 17:27:25.585916 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:25.585862 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:25.888530 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:25.888424 2560 patch_prober.go:28] interesting pod/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 18 17:27:25.888811 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:25.888782 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 18 17:27:25.923627 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:25.921416 2560 patch_prober.go:28] interesting pod/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 18 17:27:25.923627 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:25.921497 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 18 17:27:26.496305 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:26.496242 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:27:26.562586 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:26.562549 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:26.719967 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:26.719939 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:26.892518 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:26.892492 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:27:26.913884 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:26.913860 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:27:26.925208 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:26.925175 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:27:26.925596 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:26.925530 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:27:26.926132 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:26.926116 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:27:27.921460 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:27.921431 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:28.129012 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:28.128967 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:28.256378 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:28.256263 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:28.369203 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:28.369161 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:28.585988 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:28.585902 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:28.731259 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:28.731225 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:28.873382 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:28.872963 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:28.875708 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:28.875684 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v"] Sep 18 17:27:28.924483 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:28.924454 2560 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" secret="" err="secret \"e2e-ait-5cfd4b5c-openshift-default-dockercfg-nr2d5\" not found" Sep 18 17:27:29.266491 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:29.266437 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:29.408886 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:29.408855 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:29.507699 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:29.507567 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:29.931443 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:29.931398 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerName="istio-proxy" containerID="cri-o://d22367a6c00d38cf81d7d3fa33c84c54e109f47dbd24653ae1ce206e4a9d6832" gracePeriod=30 Sep 18 17:27:29.946451 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:29.946419 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f"] Sep 18 17:27:29.950110 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:29.950092 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:29.952806 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:29.952785 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fea15b54-gw-openshift-default-dockercfg-6b62j\"" Sep 18 17:27:29.992014 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:29.991971 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:29.996914 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:29.996890 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f"] Sep 18 17:27:30.053344 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.053288 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cpgg\" (UniqueName: \"kubernetes.io/projected/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-kube-api-access-8cpgg\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.053599 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.053581 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-workload-socket\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.053713 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.053700 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-workload-certs\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.053857 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.053844 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istiod-ca-cert\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.053960 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.053945 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-data\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.054069 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.054057 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-envoy\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.054162 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.054150 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-token\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.054263 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.054250 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-credential-socket\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.054556 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.054516 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-podinfo\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.113107 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.113066 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:30.156615 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.156575 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-podinfo\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.156828 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.156651 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cpgg\" (UniqueName: \"kubernetes.io/projected/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-kube-api-access-8cpgg\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.156828 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.156691 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-workload-socket\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.156828 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.156726 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-workload-certs\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.156828 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.156803 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istiod-ca-cert\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.157056 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.156833 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-data\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.157056 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.156877 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-envoy\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.157056 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.156907 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-token\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.157056 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.156938 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-credential-socket\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.157271 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.157155 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-workload-certs\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.157271 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.157239 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-workload-socket\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.157428 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.157355 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-credential-socket\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.157666 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.157644 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-data\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.158091 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.158068 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istiod-ca-cert\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.159346 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.159324 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-podinfo\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.159852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.159832 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-envoy\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.183665 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.183581 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-token\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.185906 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.185880 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cpgg\" (UniqueName: \"kubernetes.io/projected/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-kube-api-access-8cpgg\") pod \"e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.261825 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.261787 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:30.461505 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.461412 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:30.556806 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.556776 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f"] Sep 18 17:27:30.559494 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:27:30.559465 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d1e425_0bdb_4e83_9158_d7c8dbb96c55.slice/crio-8bdc381a0a2e504b1287c5c571dd5b83db4f30b842fe540918c61fd5e70de744 WatchSource:0}: Error finding container 8bdc381a0a2e504b1287c5c571dd5b83db4f30b842fe540918c61fd5e70de744: Status 404 returned error can't find the container with id 8bdc381a0a2e504b1287c5c571dd5b83db4f30b842fe540918c61fd5e70de744 Sep 18 17:27:30.562470 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.562431 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:30.562585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.562518 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:30.562649 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.562592 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:30.938719 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.938676 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" event={"ID":"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55","Type":"ContainerStarted","Data":"6435e00999140b5800e47f878595afa63a3d5bbd41ddd25c0bd68eb46ffacbd1"} Sep 18 17:27:30.938923 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.938732 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" event={"ID":"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55","Type":"ContainerStarted","Data":"8bdc381a0a2e504b1287c5c571dd5b83db4f30b842fe540918c61fd5e70de744"} Sep 18 17:27:30.964921 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:30.964840 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" podStartSLOduration=1.964825003 podStartE2EDuration="1.964825003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:30.964107919 +0000 UTC m=+764.217268765" watchObservedRunningTime="2026-09-18 17:27:30.964825003 +0000 UTC m=+764.217985819" Sep 18 17:27:31.151448 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:31.151413 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:31.262614 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:31.262515 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:31.267969 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:31.267928 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:31.945895 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:31.945836 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:31.946937 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:31.946912 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:32.056984 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:32.056940 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:32.210755 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:32.210677 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:27:32.616983 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:32.616949 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:32.718557 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:32.718470 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:32.953598 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:32.953565 2560 generic.go:356] "Generic (PLEG): container finished" podID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerID="54fae24d3592f8f51db53690294b32e2f2d4824cb3168653e1c61acceb458ca8" exitCode=0 Sep 18 17:27:32.953975 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:32.953637 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" event={"ID":"6d34c7af-3bbd-4d5a-bec3-b470994af0da","Type":"ContainerDied","Data":"54fae24d3592f8f51db53690294b32e2f2d4824cb3168653e1c61acceb458ca8"} Sep 18 17:27:32.954203 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:32.954187 2560 scope.go:117] "RemoveContainer" containerID="54fae24d3592f8f51db53690294b32e2f2d4824cb3168653e1c61acceb458ca8" Sep 18 17:27:32.955517 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:32.955488 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:32.955610 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:32.955559 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:32.955610 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:32.955592 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:33.135651 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:33.135613 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:33.970112 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:33.970073 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" event={"ID":"6d34c7af-3bbd-4d5a-bec3-b470994af0da","Type":"ContainerStarted","Data":"246af98795d4b87cb576437788f3f339ebccff43fe32f3eb11425f3d201bd0f2"} Sep 18 17:27:34.151563 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:34.151523 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:34.660496 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:34.660459 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:34.661823 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:34.661789 2560 patch_prober.go:28] interesting pod/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 18 17:27:34.661961 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:34.661872 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 18 17:27:34.835629 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:34.835599 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:35.243321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.243271 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:27:35.311458 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.311427 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-workload-socket\") pod \"ffb83f58-69fe-4648-a093-6fdd0239ba34\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " Sep 18 17:27:35.311655 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.311508 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-podinfo\") pod \"ffb83f58-69fe-4648-a093-6fdd0239ba34\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " Sep 18 17:27:35.311655 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.311542 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb83f58-69fe-4648-a093-6fdd0239ba34-istiod-ca-cert\") pod \"ffb83f58-69fe-4648-a093-6fdd0239ba34\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " Sep 18 17:27:35.311655 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.311576 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-workload-certs\") pod \"ffb83f58-69fe-4648-a093-6fdd0239ba34\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " Sep 18 17:27:35.311655 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.311648 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-token\") pod \"ffb83f58-69fe-4648-a093-6fdd0239ba34\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " Sep 18 17:27:35.311875 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.311701 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ffb83f58-69fe-4648-a093-6fdd0239ba34" (UID: "ffb83f58-69fe-4648-a093-6fdd0239ba34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:27:35.311875 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.311714 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-credential-socket\") pod \"ffb83f58-69fe-4648-a093-6fdd0239ba34\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " Sep 18 17:27:35.311875 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.311766 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-envoy\") pod \"ffb83f58-69fe-4648-a093-6fdd0239ba34\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " Sep 18 17:27:35.311875 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.311798 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgnzb\" (UniqueName: \"kubernetes.io/projected/ffb83f58-69fe-4648-a093-6fdd0239ba34-kube-api-access-jgnzb\") pod \"ffb83f58-69fe-4648-a093-6fdd0239ba34\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " Sep 18 17:27:35.311875 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.311827 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-data\") pod \"ffb83f58-69fe-4648-a093-6fdd0239ba34\" (UID: \"ffb83f58-69fe-4648-a093-6fdd0239ba34\") " Sep 18 17:27:35.312137 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.311862 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ffb83f58-69fe-4648-a093-6fdd0239ba34" (UID: "ffb83f58-69fe-4648-a093-6fdd0239ba34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:27:35.312137 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.311910 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ffb83f58-69fe-4648-a093-6fdd0239ba34" (UID: "ffb83f58-69fe-4648-a093-6fdd0239ba34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:27:35.312137 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.311941 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb83f58-69fe-4648-a093-6fdd0239ba34-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ffb83f58-69fe-4648-a093-6fdd0239ba34" (UID: "ffb83f58-69fe-4648-a093-6fdd0239ba34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:27:35.312137 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.312099 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-data" (OuterVolumeSpecName: "istio-data") pod "ffb83f58-69fe-4648-a093-6fdd0239ba34" (UID: "ffb83f58-69fe-4648-a093-6fdd0239ba34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:27:35.312368 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.312252 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffb83f58-69fe-4648-a093-6fdd0239ba34-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:27:35.312368 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.312271 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:27:35.312368 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.312284 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:27:35.312368 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.312315 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:27:35.312368 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.312328 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:27:35.314278 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.314249 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ffb83f58-69fe-4648-a093-6fdd0239ba34" (UID: "ffb83f58-69fe-4648-a093-6fdd0239ba34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:27:35.314416 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.314284 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ffb83f58-69fe-4648-a093-6fdd0239ba34" (UID: "ffb83f58-69fe-4648-a093-6fdd0239ba34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:27:35.314416 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.314326 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-token" (OuterVolumeSpecName: "istio-token") pod "ffb83f58-69fe-4648-a093-6fdd0239ba34" (UID: "ffb83f58-69fe-4648-a093-6fdd0239ba34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:27:35.314416 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.314371 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb83f58-69fe-4648-a093-6fdd0239ba34-kube-api-access-jgnzb" (OuterVolumeSpecName: "kube-api-access-jgnzb") pod "ffb83f58-69fe-4648-a093-6fdd0239ba34" (UID: "ffb83f58-69fe-4648-a093-6fdd0239ba34"). InnerVolumeSpecName "kube-api-access-jgnzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:27:35.413813 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.413777 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:27:35.413813 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.413808 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgnzb\" (UniqueName: \"kubernetes.io/projected/ffb83f58-69fe-4648-a093-6fdd0239ba34-kube-api-access-jgnzb\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:27:35.413813 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.413820 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:27:35.414056 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.413829 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffb83f58-69fe-4648-a093-6fdd0239ba34-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:27:35.661642 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.661604 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:35.667318 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.663660 2560 patch_prober.go:28] interesting pod/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 18 17:27:35.667318 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.663719 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 18 17:27:35.786203 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.786172 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:35.986268 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.986181 2560 generic.go:356] "Generic (PLEG): container finished" podID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerID="d22367a6c00d38cf81d7d3fa33c84c54e109f47dbd24653ae1ce206e4a9d6832" exitCode=0 Sep 18 17:27:35.986441 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.986321 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" event={"ID":"ffb83f58-69fe-4648-a093-6fdd0239ba34","Type":"ContainerDied","Data":"d22367a6c00d38cf81d7d3fa33c84c54e109f47dbd24653ae1ce206e4a9d6832"} Sep 18 17:27:35.986441 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.986360 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" event={"ID":"ffb83f58-69fe-4648-a093-6fdd0239ba34","Type":"ContainerDied","Data":"55b44608e6319a382bf883a9b9791ec60fac296f2e2f60d2bdb84d45a86d5131"} Sep 18 17:27:35.986441 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.986380 2560 scope.go:117] "RemoveContainer" containerID="d22367a6c00d38cf81d7d3fa33c84c54e109f47dbd24653ae1ce206e4a9d6832" Sep 18 17:27:35.986571 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:35.986495 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v" Sep 18 17:27:36.000561 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.000437 2560 scope.go:117] "RemoveContainer" containerID="02f29e9ab6b3901b37888aa096bedc317223cbdd2b47cf992b7b002d5ca9545e" Sep 18 17:27:36.012282 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.012018 2560 scope.go:117] "RemoveContainer" containerID="d22367a6c00d38cf81d7d3fa33c84c54e109f47dbd24653ae1ce206e4a9d6832" Sep 18 17:27:36.012584 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:27:36.012483 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22367a6c00d38cf81d7d3fa33c84c54e109f47dbd24653ae1ce206e4a9d6832\": container with ID starting with d22367a6c00d38cf81d7d3fa33c84c54e109f47dbd24653ae1ce206e4a9d6832 not found: ID does not exist" containerID="d22367a6c00d38cf81d7d3fa33c84c54e109f47dbd24653ae1ce206e4a9d6832" Sep 18 17:27:36.012584 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.012521 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22367a6c00d38cf81d7d3fa33c84c54e109f47dbd24653ae1ce206e4a9d6832"} err="failed to get container status \"d22367a6c00d38cf81d7d3fa33c84c54e109f47dbd24653ae1ce206e4a9d6832\": rpc error: code = NotFound desc = could not find container \"d22367a6c00d38cf81d7d3fa33c84c54e109f47dbd24653ae1ce206e4a9d6832\": container with ID starting with d22367a6c00d38cf81d7d3fa33c84c54e109f47dbd24653ae1ce206e4a9d6832 not found: ID does not exist" Sep 18 17:27:36.012584 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.012547 2560 scope.go:117] "RemoveContainer" containerID="02f29e9ab6b3901b37888aa096bedc317223cbdd2b47cf992b7b002d5ca9545e" Sep 18 17:27:36.012911 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:27:36.012887 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f29e9ab6b3901b37888aa096bedc317223cbdd2b47cf992b7b002d5ca9545e\": container with ID starting with 02f29e9ab6b3901b37888aa096bedc317223cbdd2b47cf992b7b002d5ca9545e not found: ID does not exist" containerID="02f29e9ab6b3901b37888aa096bedc317223cbdd2b47cf992b7b002d5ca9545e" Sep 18 17:27:36.012984 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.012916 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f29e9ab6b3901b37888aa096bedc317223cbdd2b47cf992b7b002d5ca9545e"} err="failed to get container status \"02f29e9ab6b3901b37888aa096bedc317223cbdd2b47cf992b7b002d5ca9545e\": rpc error: code = NotFound desc = could not find container \"02f29e9ab6b3901b37888aa096bedc317223cbdd2b47cf992b7b002d5ca9545e\": container with ID starting with 02f29e9ab6b3901b37888aa096bedc317223cbdd2b47cf992b7b002d5ca9545e not found: ID does not exist" Sep 18 17:27:36.017709 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.017662 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v"] Sep 18 17:27:36.020673 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.020642 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5cfd4b5c-openshift-default-7f96c5556-m5n6v"] Sep 18 17:27:36.057687 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.057650 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:36.177288 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.177207 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:36.275710 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.275563 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:36.570496 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.570387 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:36.661482 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.661439 2560 patch_prober.go:28] interesting pod/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 18 17:27:36.661691 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.661523 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 18 17:27:36.734885 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.734854 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:36.870922 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:36.870820 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" path="/var/lib/kubelet/pods/ffb83f58-69fe-4648-a093-6fdd0239ba34/volumes" Sep 18 17:27:37.660581 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:37.660542 2560 patch_prober.go:28] interesting pod/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 18 17:27:37.661203 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:37.661169 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 18 17:27:37.772226 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:37.772171 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:38.661377 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:38.661328 2560 patch_prober.go:28] interesting pod/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 18 17:27:38.661889 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:38.661414 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 18 17:27:38.729133 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:38.729097 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:39.660591 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:39.660556 2560 patch_prober.go:28] interesting pod/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 18 17:27:39.660765 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:39.660618 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 18 17:27:40.665186 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:40.665154 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:40.666267 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:40.666245 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:27:50.917267 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:50.917234 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:27:53.106947 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:53.106915 2560 generic.go:356] "Generic (PLEG): container finished" podID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerID="6435e00999140b5800e47f878595afa63a3d5bbd41ddd25c0bd68eb46ffacbd1" exitCode=0 Sep 18 17:27:53.107391 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:53.106987 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" event={"ID":"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55","Type":"ContainerDied","Data":"6435e00999140b5800e47f878595afa63a3d5bbd41ddd25c0bd68eb46ffacbd1"} Sep 18 17:27:53.107521 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:53.107505 2560 scope.go:117] "RemoveContainer" containerID="6435e00999140b5800e47f878595afa63a3d5bbd41ddd25c0bd68eb46ffacbd1" Sep 18 17:27:53.108672 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:53.108642 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:53.108735 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:53.108711 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:53.108774 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:53.108744 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:27:54.114865 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:54.114835 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" event={"ID":"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55","Type":"ContainerStarted","Data":"761d98b54f1aef2ce6a2a8ef27fcfbff42515708ff017cb644cf81f9588c7d23"} Sep 18 17:27:54.262561 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:54.262520 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:54.262951 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:54.262911 2560 patch_prober.go:28] interesting pod/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 18 17:27:54.263052 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:54.262974 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 18 17:27:55.262754 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:55.262702 2560 patch_prober.go:28] interesting pod/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 18 17:27:55.263314 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:55.262780 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 18 17:27:56.263901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:56.263859 2560 patch_prober.go:28] interesting pod/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 18 17:27:56.264535 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:56.264486 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 18 17:27:57.262860 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:57.262821 2560 patch_prober.go:28] interesting pod/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 18 17:27:57.263287 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:57.263246 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 18 17:27:58.078809 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:58.078772 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:58.262373 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:58.262332 2560 patch_prober.go:28] interesting pod/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 18 17:27:58.262582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:58.262419 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 18 17:27:58.825724 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:58.825691 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:27:59.363752 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:59.363723 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:59.364248 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:59.364138 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:59.365054 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:59.365026 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:27:59.757441 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:27:59.757410 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:00.700531 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:00.700498 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:01.483632 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:01.483598 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:02.470643 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:02.470611 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:03.197268 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:03.197227 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:03.733190 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:03.733160 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:04.532654 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:04.532617 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:05.669308 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:05.669268 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:06.799022 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:06.798979 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:07.967394 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:07.967366 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:08.820917 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:08.820880 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:09.757266 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:09.757233 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:13.427828 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:13.427791 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:13.992898 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:13.992868 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:14.053560 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:14.053511 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:14.117436 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:14.117408 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:14.199170 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:14.199133 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:14.259601 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:14.259523 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:14.325392 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:14.325356 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f"] Sep 18 17:28:14.325705 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:14.325670 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerName="istio-proxy" containerID="cri-o://761d98b54f1aef2ce6a2a8ef27fcfbff42515708ff017cb644cf81f9588c7d23" gracePeriod=30 Sep 18 17:28:14.366167 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:14.366123 2560 patch_prober.go:28] interesting pod/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 17:28:14.366340 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:14.366192 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 17:28:14.369277 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:14.369249 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:14.743047 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:14.743017 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:14.823040 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:14.823008 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:15.230101 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:15.230064 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:15.654573 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:15.654526 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:15.757178 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:15.757148 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:16.856823 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:16.856789 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:17.915465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:17.915430 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:18.057800 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:18.057768 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:18.191030 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:18.190947 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:18.743005 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:18.742977 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:19.040329 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.040229 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:19.591714 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.591652 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:28:19.723277 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.723243 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-credential-socket\") pod \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " Sep 18 17:28:19.723477 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.723346 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cpgg\" (UniqueName: \"kubernetes.io/projected/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-kube-api-access-8cpgg\") pod \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " Sep 18 17:28:19.723477 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.723376 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-workload-certs\") pod \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " Sep 18 17:28:19.723477 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.723403 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-podinfo\") pod \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " Sep 18 17:28:19.723477 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.723436 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-token\") pod \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " Sep 18 17:28:19.723708 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.723509 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" (UID: "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:28:19.723708 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.723521 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-workload-socket\") pod \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " Sep 18 17:28:19.723708 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.723602 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-envoy\") pod \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " Sep 18 17:28:19.723708 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.723626 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istiod-ca-cert\") pod \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " Sep 18 17:28:19.723921 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.723712 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-data\") pod \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\" (UID: \"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55\") " Sep 18 17:28:19.723921 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.723718 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" (UID: "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:28:19.723921 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.723804 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" (UID: "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:28:19.724078 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.724053 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-data" (OuterVolumeSpecName: "istio-data") pod "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" (UID: "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:28:19.724134 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.724089 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" (UID: "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:28:19.724204 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.724181 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:28:19.724264 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.724204 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:28:19.724264 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.724219 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:28:19.724264 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.724232 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:28:19.724264 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.724247 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:28:19.725676 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.725653 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" (UID: "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:28:19.725782 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.725670 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-kube-api-access-8cpgg" (OuterVolumeSpecName: "kube-api-access-8cpgg") pod "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" (UID: "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55"). InnerVolumeSpecName "kube-api-access-8cpgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:28:19.726043 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.726024 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" (UID: "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:28:19.726100 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.726023 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" (UID: "d3d1e425-0bdb-4e83-9158-d7c8dbb96c55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:28:19.824885 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.824850 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cpgg\" (UniqueName: \"kubernetes.io/projected/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-kube-api-access-8cpgg\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:28:19.824885 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.824884 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:28:19.824885 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.824896 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:28:19.825132 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:19.824907 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:28:20.278420 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.278388 2560 generic.go:356] "Generic (PLEG): container finished" podID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerID="761d98b54f1aef2ce6a2a8ef27fcfbff42515708ff017cb644cf81f9588c7d23" exitCode=0 Sep 18 17:28:20.278894 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.278454 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" Sep 18 17:28:20.278894 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.278484 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" event={"ID":"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55","Type":"ContainerDied","Data":"761d98b54f1aef2ce6a2a8ef27fcfbff42515708ff017cb644cf81f9588c7d23"} Sep 18 17:28:20.278894 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.278547 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f" event={"ID":"d3d1e425-0bdb-4e83-9158-d7c8dbb96c55","Type":"ContainerDied","Data":"8bdc381a0a2e504b1287c5c571dd5b83db4f30b842fe540918c61fd5e70de744"} Sep 18 17:28:20.278894 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.278575 2560 scope.go:117] "RemoveContainer" containerID="761d98b54f1aef2ce6a2a8ef27fcfbff42515708ff017cb644cf81f9588c7d23" Sep 18 17:28:20.288980 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.288962 2560 scope.go:117] "RemoveContainer" containerID="6435e00999140b5800e47f878595afa63a3d5bbd41ddd25c0bd68eb46ffacbd1" Sep 18 17:28:20.297656 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.297633 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:20.298284 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.298269 2560 scope.go:117] "RemoveContainer" containerID="761d98b54f1aef2ce6a2a8ef27fcfbff42515708ff017cb644cf81f9588c7d23" Sep 18 17:28:20.298599 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:28:20.298581 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761d98b54f1aef2ce6a2a8ef27fcfbff42515708ff017cb644cf81f9588c7d23\": container with ID starting with 761d98b54f1aef2ce6a2a8ef27fcfbff42515708ff017cb644cf81f9588c7d23 not found: ID does not exist" containerID="761d98b54f1aef2ce6a2a8ef27fcfbff42515708ff017cb644cf81f9588c7d23" Sep 18 17:28:20.298651 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.298609 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761d98b54f1aef2ce6a2a8ef27fcfbff42515708ff017cb644cf81f9588c7d23"} err="failed to get container status \"761d98b54f1aef2ce6a2a8ef27fcfbff42515708ff017cb644cf81f9588c7d23\": rpc error: code = NotFound desc = could not find container \"761d98b54f1aef2ce6a2a8ef27fcfbff42515708ff017cb644cf81f9588c7d23\": container with ID starting with 761d98b54f1aef2ce6a2a8ef27fcfbff42515708ff017cb644cf81f9588c7d23 not found: ID does not exist" Sep 18 17:28:20.298651 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.298626 2560 scope.go:117] "RemoveContainer" containerID="6435e00999140b5800e47f878595afa63a3d5bbd41ddd25c0bd68eb46ffacbd1" Sep 18 17:28:20.298866 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:28:20.298849 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6435e00999140b5800e47f878595afa63a3d5bbd41ddd25c0bd68eb46ffacbd1\": container with ID starting with 6435e00999140b5800e47f878595afa63a3d5bbd41ddd25c0bd68eb46ffacbd1 not found: ID does not exist" containerID="6435e00999140b5800e47f878595afa63a3d5bbd41ddd25c0bd68eb46ffacbd1" Sep 18 17:28:20.298915 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.298875 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6435e00999140b5800e47f878595afa63a3d5bbd41ddd25c0bd68eb46ffacbd1"} err="failed to get container status \"6435e00999140b5800e47f878595afa63a3d5bbd41ddd25c0bd68eb46ffacbd1\": rpc error: code = NotFound desc = could not find container \"6435e00999140b5800e47f878595afa63a3d5bbd41ddd25c0bd68eb46ffacbd1\": container with ID starting with 6435e00999140b5800e47f878595afa63a3d5bbd41ddd25c0bd68eb46ffacbd1 not found: ID does not exist" Sep 18 17:28:20.317271 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.317241 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f"] Sep 18 17:28:20.321715 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.321676 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fea15b54-gw-openshift-default-77dff6d4d-m6q8f"] Sep 18 17:28:20.360096 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.360061 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:28:20.865872 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:20.865835 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" path="/var/lib/kubelet/pods/d3d1e425-0bdb-4e83-9158-d7c8dbb96c55/volumes" Sep 18 17:28:21.221308 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:21.221256 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:22.106122 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:22.106091 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:22.177833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:22.177803 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:22.478159 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:22.478128 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:22.756822 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:22.756744 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:22.875982 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:22.875947 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:23.561337 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:23.561306 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:24.325620 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:24.325588 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:25.191881 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:25.191851 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:25.926187 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:25.926148 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:35.223983 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:35.223944 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:28:40.521600 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:40.521567 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:44.745537 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:44.745501 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:45.060436 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:45.060342 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:45.832013 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:45.831972 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:46.521421 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:46.521389 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:47.406765 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:47.406733 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:48.261961 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.261926 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:48.760612 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.760577 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc"] Sep 18 17:28:48.761267 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.761246 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerName="istio-proxy" Sep 18 17:28:48.761403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.761270 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerName="istio-proxy" Sep 18 17:28:48.761403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.761307 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerName="istio-proxy" Sep 18 17:28:48.761403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.761315 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerName="istio-proxy" Sep 18 17:28:48.761403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.761327 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerName="istio-proxy" Sep 18 17:28:48.761403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.761338 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerName="istio-proxy" Sep 18 17:28:48.761403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.761355 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerName="istio-proxy" Sep 18 17:28:48.761403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.761363 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerName="istio-proxy" Sep 18 17:28:48.761748 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.761507 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerName="istio-proxy" Sep 18 17:28:48.761748 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.761521 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerName="istio-proxy" Sep 18 17:28:48.761748 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.761534 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb83f58-69fe-4648-a093-6fdd0239ba34" containerName="istio-proxy" Sep 18 17:28:48.761748 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.761550 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3d1e425-0bdb-4e83-9158-d7c8dbb96c55" containerName="istio-proxy" Sep 18 17:28:48.764079 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.764059 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc" Sep 18 17:28:48.766648 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.766629 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\"" Sep 18 17:28:48.774047 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.774022 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc"] Sep 18 17:28:48.836183 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.836145 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1fa82f-1fec-4059-8d6d-80e6f359a5f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8bd53078-757xc\" (UID: \"8f1fa82f-1fec-4059-8d6d-80e6f359a5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc" Sep 18 17:28:48.937526 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.937482 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1fa82f-1fec-4059-8d6d-80e6f359a5f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8bd53078-757xc\" (UID: \"8f1fa82f-1fec-4059-8d6d-80e6f359a5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc" Sep 18 17:28:48.938110 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:48.938089 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1fa82f-1fec-4059-8d6d-80e6f359a5f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8bd53078-757xc\" (UID: \"8f1fa82f-1fec-4059-8d6d-80e6f359a5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc" Sep 18 17:28:49.076578 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:49.076477 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc" Sep 18 17:28:49.232488 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:49.232454 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:49.356527 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:49.356498 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc"] Sep 18 17:28:49.359431 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:28:49.359401 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1fa82f_1fec_4059_8d6d_80e6f359a5f6.slice/crio-fee52fde07dd1c1de3e5a248e4fa7272d92db9e865315296ff1bef752ac4c3bc WatchSource:0}: Error finding container fee52fde07dd1c1de3e5a248e4fa7272d92db9e865315296ff1bef752ac4c3bc: Status 404 returned error can't find the container with id fee52fde07dd1c1de3e5a248e4fa7272d92db9e865315296ff1bef752ac4c3bc Sep 18 17:28:49.453448 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:49.453413 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc" event={"ID":"8f1fa82f-1fec-4059-8d6d-80e6f359a5f6","Type":"ContainerStarted","Data":"fee52fde07dd1c1de3e5a248e4fa7272d92db9e865315296ff1bef752ac4c3bc"} Sep 18 17:28:50.145788 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:50.145745 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:28:50.190834 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:50.190800 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:50.979789 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:50.979752 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:51.914627 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:51.914596 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:52.480473 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:52.480438 2560 generic.go:356] "Generic (PLEG): container finished" podID="8f1fa82f-1fec-4059-8d6d-80e6f359a5f6" containerID="eec1f0979419f96d85b63d49733b08e45a9893af91278527df3f9b900b9043f5" exitCode=0 Sep 18 17:28:52.480649 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:52.480518 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc" event={"ID":"8f1fa82f-1fec-4059-8d6d-80e6f359a5f6","Type":"ContainerDied","Data":"eec1f0979419f96d85b63d49733b08e45a9893af91278527df3f9b900b9043f5"} Sep 18 17:28:52.992961 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:52.992933 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:53.618215 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:53.618195 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc" Sep 18 17:28:53.683682 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:53.683652 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:53.793646 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:53.793562 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1fa82f-1fec-4059-8d6d-80e6f359a5f6-maas-api-service-ca\") pod \"8f1fa82f-1fec-4059-8d6d-80e6f359a5f6\" (UID: \"8f1fa82f-1fec-4059-8d6d-80e6f359a5f6\") " Sep 18 17:28:53.793875 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:53.793853 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f1fa82f-1fec-4059-8d6d-80e6f359a5f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f1fa82f-1fec-4059-8d6d-80e6f359a5f6" (UID: "8f1fa82f-1fec-4059-8d6d-80e6f359a5f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:28:53.894807 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:53.894775 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f1fa82f-1fec-4059-8d6d-80e6f359a5f6-maas-api-service-ca\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:28:54.495055 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:54.495027 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc" Sep 18 17:28:54.495447 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:54.495062 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc" event={"ID":"8f1fa82f-1fec-4059-8d6d-80e6f359a5f6","Type":"ContainerDied","Data":"fee52fde07dd1c1de3e5a248e4fa7272d92db9e865315296ff1bef752ac4c3bc"} Sep 18 17:28:54.495447 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:54.495104 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fee52fde07dd1c1de3e5a248e4fa7272d92db9e865315296ff1bef752ac4c3bc" Sep 18 17:28:54.540898 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:54.540867 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:57.735470 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:57.735444 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:28:59.502347 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:28:59.502285 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:00.085990 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:00.085961 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:00.184017 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:00.183987 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:00.266648 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:00.266618 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:00.395870 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:00.395780 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:00.436914 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:00.436868 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc"] Sep 18 17:29:00.442128 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:00.442102 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8bd53078-757xc"] Sep 18 17:29:00.497890 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:00.497863 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:00.865960 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:00.865927 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f1fa82f-1fec-4059-8d6d-80e6f359a5f6" path="/var/lib/kubelet/pods/8f1fa82f-1fec-4059-8d6d-80e6f359a5f6/volumes" Sep 18 17:29:00.930427 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:00.930390 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:01.699689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:01.699655 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:02.093277 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.093203 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:02.267809 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.267776 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:02.383145 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.383064 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb"] Sep 18 17:29:02.383557 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.383544 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f1fa82f-1fec-4059-8d6d-80e6f359a5f6" containerName="revoke-keys" Sep 18 17:29:02.383632 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.383559 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f1fa82f-1fec-4059-8d6d-80e6f359a5f6" containerName="revoke-keys" Sep 18 17:29:02.383677 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.383639 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f1fa82f-1fec-4059-8d6d-80e6f359a5f6" containerName="revoke-keys" Sep 18 17:29:02.385876 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.385861 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.388929 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.388912 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-fb1cf26a-openshift-default-dockercfg-nmn94\"" Sep 18 17:29:02.400738 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.400718 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb"] Sep 18 17:29:02.420389 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.420369 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:02.580934 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.580902 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-podinfo\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.581088 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.580982 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-workload-certs\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.581088 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.581019 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smgf\" (UniqueName: \"kubernetes.io/projected/79bea44d-81c3-44bd-b905-fea0e2181f50-kube-api-access-9smgf\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.581180 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.581085 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-workload-socket\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.581180 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.581135 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bea44d-81c3-44bd-b905-fea0e2181f50-istiod-ca-cert\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.581243 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.581203 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-token\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.581243 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.581228 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-data\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.581355 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.581338 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-envoy\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.581395 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.581366 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-credential-socket\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.589194 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.589167 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:02.682158 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682129 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-workload-socket\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.682383 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682170 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bea44d-81c3-44bd-b905-fea0e2181f50-istiod-ca-cert\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.682383 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682312 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-token\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.682383 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682375 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-data\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.682568 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682449 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-envoy\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.682568 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682475 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-credential-socket\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.682568 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682517 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-podinfo\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.682732 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682587 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-workload-certs\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.682732 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682622 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smgf\" (UniqueName: \"kubernetes.io/projected/79bea44d-81c3-44bd-b905-fea0e2181f50-kube-api-access-9smgf\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.682829 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682725 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-data\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.682915 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682893 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bea44d-81c3-44bd-b905-fea0e2181f50-istiod-ca-cert\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.683043 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682893 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-credential-socket\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.683043 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682952 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-workload-certs\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.683043 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.682979 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-workload-socket\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.684713 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.684686 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-envoy\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.684925 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.684904 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-podinfo\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.691057 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.691036 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-token\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.691249 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.691232 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smgf\" (UniqueName: \"kubernetes.io/projected/79bea44d-81c3-44bd-b905-fea0e2181f50-kube-api-access-9smgf\") pod \"e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.696106 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.696087 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:02.981143 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.981110 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb"] Sep 18 17:29:02.981344 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:29:02.981282 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bea44d_81c3_44bd_b905_fea0e2181f50.slice/crio-892c0178efb851577624723b9fff11d606fdceb11da707f32062141358856fc9 WatchSource:0}: Error finding container 892c0178efb851577624723b9fff11d606fdceb11da707f32062141358856fc9: Status 404 returned error can't find the container with id 892c0178efb851577624723b9fff11d606fdceb11da707f32062141358856fc9 Sep 18 17:29:02.983822 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.983779 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:29:02.983907 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.983872 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:29:02.983949 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:02.983917 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:29:03.109364 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:03.109285 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:03.552497 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:03.552462 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" event={"ID":"79bea44d-81c3-44bd-b905-fea0e2181f50","Type":"ContainerStarted","Data":"e7a6e2f2bb5f58e4f258285cb7b2d428d30f6b77e193ef7a734f3ef49635004c"} Sep 18 17:29:03.552673 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:03.552547 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" event={"ID":"79bea44d-81c3-44bd-b905-fea0e2181f50","Type":"ContainerStarted","Data":"892c0178efb851577624723b9fff11d606fdceb11da707f32062141358856fc9"} Sep 18 17:29:03.580927 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:03.580875 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" podStartSLOduration=1.580858833 podStartE2EDuration="1.580858833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:29:03.579248684 +0000 UTC m=+856.832409512" watchObservedRunningTime="2026-09-18 17:29:03.580858833 +0000 UTC m=+856.834019647" Sep 18 17:29:03.696507 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:03.696478 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:04.075154 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:04.075117 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:04.701680 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:04.701653 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:05.123565 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:05.123492 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:05.216050 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:05.216019 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:05.564086 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:05.564054 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:05.565130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:05.565110 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:05.978653 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:05.978611 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:06.963551 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:06.963524 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz"] Sep 18 17:29:06.966282 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:06.966267 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:06.976072 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:06.976051 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-fb1cf26a-serving-cert\"" Sep 18 17:29:07.002887 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.002752 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz"] Sep 18 17:29:07.003599 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.003571 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:07.032211 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.032189 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbpvq\" (UniqueName: \"kubernetes.io/projected/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-kube-api-access-lbpvq\") pod \"maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:07.032360 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.032306 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-openshift-service-ca\") pod \"maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:07.032360 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.032327 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-maas-api-tls\") pod \"maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:07.032437 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.032405 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-metrics-certs\") pod \"maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:07.133034 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.132998 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbpvq\" (UniqueName: \"kubernetes.io/projected/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-kube-api-access-lbpvq\") pod \"maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:07.133171 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.133093 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-openshift-service-ca\") pod \"maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:07.133171 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.133111 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-maas-api-tls\") pod \"maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:07.133171 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.133146 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-metrics-certs\") pod \"maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:07.133743 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.133722 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-openshift-service-ca\") pod \"maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:07.135544 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.135521 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-metrics-certs\") pod \"maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:07.135544 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.135537 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-maas-api-tls\") pod \"maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:07.148483 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.148457 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbpvq\" (UniqueName: \"kubernetes.io/projected/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-kube-api-access-lbpvq\") pod \"maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:07.154800 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.154762 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:07.276902 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.276825 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:07.548240 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.548216 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz"] Sep 18 17:29:07.549996 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:29:07.549970 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9fe9d3_27ac_4c01_bbb5_7d1373fd4e33.slice/crio-7f912fd94cc5f2f3ed34c0b27eb5440c228f3a3e7cdef3bd37b44c45383d7863 WatchSource:0}: Error finding container 7f912fd94cc5f2f3ed34c0b27eb5440c228f3a3e7cdef3bd37b44c45383d7863: Status 404 returned error can't find the container with id 7f912fd94cc5f2f3ed34c0b27eb5440c228f3a3e7cdef3bd37b44c45383d7863 Sep 18 17:29:07.577085 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.577052 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" event={"ID":"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33","Type":"ContainerStarted","Data":"7f912fd94cc5f2f3ed34c0b27eb5440c228f3a3e7cdef3bd37b44c45383d7863"} Sep 18 17:29:07.615380 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.615348 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:07.716718 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:07.716687 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:08.043829 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:08.043779 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:08.584594 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:08.584506 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" event={"ID":"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33","Type":"ContainerStarted","Data":"849a62f349c6b343e84a497cec5046cdb32ad8ee51f4dff4525a29de1665c4d0"} Sep 18 17:29:08.584753 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:08.584638 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:08.993991 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:08.993959 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:09.867365 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:09.867329 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:10.638932 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:10.638901 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:11.518514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:11.518485 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:14.596438 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:14.596411 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:14.620597 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:14.620545 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" podStartSLOduration=7.877115632 podStartE2EDuration="8.62052985s" totalImagesPullingTime="743.414218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:29:08.636525861 +0000 UTC m=+861.889686671" watchObservedRunningTime="2026-09-18 17:29:14.62052985 +0000 UTC m=+867.873690725" Sep 18 17:29:17.613419 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:17.613387 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:29:20.456266 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:20.456207 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:21.352210 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:21.352162 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:22.229193 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:22.229157 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:22.981162 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:22.981096 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:23.684015 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:23.683982 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:24.388217 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:24.388183 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:25.452657 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:25.452621 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:26.344969 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:26.344941 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:27.215093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:27.215049 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:27.717554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:27.717513 2560 generic.go:356] "Generic (PLEG): container finished" podID="79bea44d-81c3-44bd-b905-fea0e2181f50" containerID="e7a6e2f2bb5f58e4f258285cb7b2d428d30f6b77e193ef7a734f3ef49635004c" exitCode=0 Sep 18 17:29:27.717768 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:27.717572 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" event={"ID":"79bea44d-81c3-44bd-b905-fea0e2181f50","Type":"ContainerDied","Data":"e7a6e2f2bb5f58e4f258285cb7b2d428d30f6b77e193ef7a734f3ef49635004c"} Sep 18 17:29:27.718092 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:27.718054 2560 scope.go:117] "RemoveContainer" containerID="e7a6e2f2bb5f58e4f258285cb7b2d428d30f6b77e193ef7a734f3ef49635004c" Sep 18 17:29:27.719489 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:27.719445 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:29:27.719563 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:27.719527 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:29:27.719605 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:27.719560 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:29:27.971666 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:27.971516 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:28.724722 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:28.724693 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" event={"ID":"79bea44d-81c3-44bd-b905-fea0e2181f50","Type":"ContainerStarted","Data":"828d3d9dae0550b3d348da811d4dde5adb4a2f77514faf8f73717645836595b7"} Sep 18 17:29:28.837616 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:28.837583 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:29.696930 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:29.696898 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:29.697161 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:29.697143 2560 patch_prober.go:28] interesting pod/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 18 17:29:29.697208 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:29.697185 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" podUID="79bea44d-81c3-44bd-b905-fea0e2181f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 18 17:29:29.972012 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:29.971849 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:30.696493 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:30.696450 2560 patch_prober.go:28] interesting pod/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 18 17:29:30.696672 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:30.696528 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" podUID="79bea44d-81c3-44bd-b905-fea0e2181f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 18 17:29:31.073600 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:31.073460 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:31.696655 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:31.696617 2560 patch_prober.go:28] interesting pod/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 18 17:29:31.696846 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:31.696680 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" podUID="79bea44d-81c3-44bd-b905-fea0e2181f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 18 17:29:31.926090 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:31.926031 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:32.630543 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:32.630510 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:32.696929 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:32.696763 2560 patch_prober.go:28] interesting pod/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Sep 18 17:29:32.696929 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:32.696842 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" podUID="79bea44d-81c3-44bd-b905-fea0e2181f50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Sep 18 17:29:32.734214 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:32.734190 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:32.831528 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:32.831492 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:32.917540 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:32.917513 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:32.988504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:32.988473 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:33.307022 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:33.306936 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:33.700366 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:33.700334 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:33.700747 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:33.700609 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:33.701158 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:33.701142 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:34.166774 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:34.166743 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:35.347604 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:35.347574 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:36.345242 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:36.345206 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:37.160088 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:37.160042 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:37.231657 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:37.231620 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:37.935888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:37.935859 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:38.353522 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.353441 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:38.538489 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.538453 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz"] Sep 18 17:29:38.538717 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.538693 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" podUID="1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33" containerName="maas-api" containerID="cri-o://849a62f349c6b343e84a497cec5046cdb32ad8ee51f4dff4525a29de1665c4d0" gracePeriod=30 Sep 18 17:29:38.563536 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.563509 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:38.666193 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.666160 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:38.763913 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.763891 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:38.784956 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.784937 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:38.792668 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.792646 2560 generic.go:356] "Generic (PLEG): container finished" podID="1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33" containerID="849a62f349c6b343e84a497cec5046cdb32ad8ee51f4dff4525a29de1665c4d0" exitCode=0 Sep 18 17:29:38.792764 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.792678 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" Sep 18 17:29:38.792764 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.792720 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" event={"ID":"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33","Type":"ContainerDied","Data":"849a62f349c6b343e84a497cec5046cdb32ad8ee51f4dff4525a29de1665c4d0"} Sep 18 17:29:38.792764 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.792760 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz" event={"ID":"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33","Type":"ContainerDied","Data":"7f912fd94cc5f2f3ed34c0b27eb5440c228f3a3e7cdef3bd37b44c45383d7863"} Sep 18 17:29:38.792894 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.792782 2560 scope.go:117] "RemoveContainer" containerID="849a62f349c6b343e84a497cec5046cdb32ad8ee51f4dff4525a29de1665c4d0" Sep 18 17:29:38.801760 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.801742 2560 scope.go:117] "RemoveContainer" containerID="849a62f349c6b343e84a497cec5046cdb32ad8ee51f4dff4525a29de1665c4d0" Sep 18 17:29:38.802020 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:29:38.801999 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"849a62f349c6b343e84a497cec5046cdb32ad8ee51f4dff4525a29de1665c4d0\": container with ID starting with 849a62f349c6b343e84a497cec5046cdb32ad8ee51f4dff4525a29de1665c4d0 not found: ID does not exist" containerID="849a62f349c6b343e84a497cec5046cdb32ad8ee51f4dff4525a29de1665c4d0" Sep 18 17:29:38.802084 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.802029 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849a62f349c6b343e84a497cec5046cdb32ad8ee51f4dff4525a29de1665c4d0"} err="failed to get container status \"849a62f349c6b343e84a497cec5046cdb32ad8ee51f4dff4525a29de1665c4d0\": rpc error: code = NotFound desc = could not find container \"849a62f349c6b343e84a497cec5046cdb32ad8ee51f4dff4525a29de1665c4d0\": container with ID starting with 849a62f349c6b343e84a497cec5046cdb32ad8ee51f4dff4525a29de1665c4d0 not found: ID does not exist" Sep 18 17:29:38.864574 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.864552 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:38.883029 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.883007 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbpvq\" (UniqueName: \"kubernetes.io/projected/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-kube-api-access-lbpvq\") pod \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " Sep 18 17:29:38.883134 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.883070 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-maas-api-tls\") pod \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " Sep 18 17:29:38.883194 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.883151 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-openshift-service-ca\") pod \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " Sep 18 17:29:38.883194 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.883166 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-metrics-certs\") pod \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\" (UID: \"1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33\") " Sep 18 17:29:38.883919 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.883490 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33" (UID: "1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:29:38.883919 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.883619 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-openshift-service-ca\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:29:38.885547 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.885519 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33" (UID: "1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:29:38.885643 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.885586 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-kube-api-access-lbpvq" (OuterVolumeSpecName: "kube-api-access-lbpvq") pod "1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33" (UID: "1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33"). InnerVolumeSpecName "kube-api-access-lbpvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:29:38.885814 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.885799 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33" (UID: "1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:29:38.920799 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.920729 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb"] Sep 18 17:29:38.920946 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.920913 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" podUID="79bea44d-81c3-44bd-b905-fea0e2181f50" containerName="istio-proxy" containerID="cri-o://828d3d9dae0550b3d348da811d4dde5adb4a2f77514faf8f73717645836595b7" gracePeriod=30 Sep 18 17:29:38.952420 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.952379 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:38.984498 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.984473 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbpvq\" (UniqueName: \"kubernetes.io/projected/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-kube-api-access-lbpvq\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:29:38.984498 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.984493 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-maas-api-tls\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:29:38.984498 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:38.984501 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33-metrics-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:29:39.116915 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:39.116890 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz"] Sep 18 17:29:39.120756 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:39.120732 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-fb1cf26a-6cc6ff867b-ccbvz"] Sep 18 17:29:39.581507 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:39.581447 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:39.707444 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:39.707411 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:39.810407 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:39.810376 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:40.068024 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:40.067990 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:40.865975 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:40.865940 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33" path="/var/lib/kubelet/pods/1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33/volumes" Sep 18 17:29:41.370801 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:41.370766 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:41.472008 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:41.471960 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:42.263759 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:42.263726 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:42.475375 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:42.475346 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:43.157284 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:43.157254 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:44.198983 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.198926 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:44.222841 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.222808 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:29:44.243173 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243144 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-envoy\") pod \"79bea44d-81c3-44bd-b905-fea0e2181f50\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " Sep 18 17:29:44.243365 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243211 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bea44d-81c3-44bd-b905-fea0e2181f50-istiod-ca-cert\") pod \"79bea44d-81c3-44bd-b905-fea0e2181f50\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " Sep 18 17:29:44.243365 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243235 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-workload-socket\") pod \"79bea44d-81c3-44bd-b905-fea0e2181f50\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " Sep 18 17:29:44.243365 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243347 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-token\") pod \"79bea44d-81c3-44bd-b905-fea0e2181f50\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " Sep 18 17:29:44.243540 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243391 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9smgf\" (UniqueName: \"kubernetes.io/projected/79bea44d-81c3-44bd-b905-fea0e2181f50-kube-api-access-9smgf\") pod \"79bea44d-81c3-44bd-b905-fea0e2181f50\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " Sep 18 17:29:44.243540 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243416 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-workload-certs\") pod \"79bea44d-81c3-44bd-b905-fea0e2181f50\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " Sep 18 17:29:44.243540 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243444 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-podinfo\") pod \"79bea44d-81c3-44bd-b905-fea0e2181f50\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " Sep 18 17:29:44.243540 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243523 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-credential-socket\") pod \"79bea44d-81c3-44bd-b905-fea0e2181f50\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " Sep 18 17:29:44.243738 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243552 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-data\") pod \"79bea44d-81c3-44bd-b905-fea0e2181f50\" (UID: \"79bea44d-81c3-44bd-b905-fea0e2181f50\") " Sep 18 17:29:44.243738 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243562 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "79bea44d-81c3-44bd-b905-fea0e2181f50" (UID: "79bea44d-81c3-44bd-b905-fea0e2181f50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:29:44.243738 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243607 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bea44d-81c3-44bd-b905-fea0e2181f50-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "79bea44d-81c3-44bd-b905-fea0e2181f50" (UID: "79bea44d-81c3-44bd-b905-fea0e2181f50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:29:44.243893 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243736 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "79bea44d-81c3-44bd-b905-fea0e2181f50" (UID: "79bea44d-81c3-44bd-b905-fea0e2181f50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:29:44.243893 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243882 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "79bea44d-81c3-44bd-b905-fea0e2181f50" (UID: "79bea44d-81c3-44bd-b905-fea0e2181f50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:29:44.244006 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243979 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:29:44.244006 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.243997 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:29:44.244110 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.244011 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bea44d-81c3-44bd-b905-fea0e2181f50-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:29:44.244110 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.244024 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:29:44.245858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.245822 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-data" (OuterVolumeSpecName: "istio-data") pod "79bea44d-81c3-44bd-b905-fea0e2181f50" (UID: "79bea44d-81c3-44bd-b905-fea0e2181f50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:29:44.246645 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.246621 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bea44d-81c3-44bd-b905-fea0e2181f50-kube-api-access-9smgf" (OuterVolumeSpecName: "kube-api-access-9smgf") pod "79bea44d-81c3-44bd-b905-fea0e2181f50" (UID: "79bea44d-81c3-44bd-b905-fea0e2181f50"). InnerVolumeSpecName "kube-api-access-9smgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:29:44.246760 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.246646 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-token" (OuterVolumeSpecName: "istio-token") pod "79bea44d-81c3-44bd-b905-fea0e2181f50" (UID: "79bea44d-81c3-44bd-b905-fea0e2181f50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:29:44.246817 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.246794 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "79bea44d-81c3-44bd-b905-fea0e2181f50" (UID: "79bea44d-81c3-44bd-b905-fea0e2181f50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:29:44.247004 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.246984 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "79bea44d-81c3-44bd-b905-fea0e2181f50" (UID: "79bea44d-81c3-44bd-b905-fea0e2181f50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:29:44.344968 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.344931 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:29:44.344968 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.344963 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:29:44.344968 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.344976 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9smgf\" (UniqueName: \"kubernetes.io/projected/79bea44d-81c3-44bd-b905-fea0e2181f50-kube-api-access-9smgf\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:29:44.345276 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.344988 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:29:44.345276 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.345001 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bea44d-81c3-44bd-b905-fea0e2181f50-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:29:44.831250 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.831220 2560 generic.go:356] "Generic (PLEG): container finished" podID="79bea44d-81c3-44bd-b905-fea0e2181f50" containerID="828d3d9dae0550b3d348da811d4dde5adb4a2f77514faf8f73717645836595b7" exitCode=0 Sep 18 17:29:44.831463 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.831268 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" Sep 18 17:29:44.831463 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.831333 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" event={"ID":"79bea44d-81c3-44bd-b905-fea0e2181f50","Type":"ContainerDied","Data":"828d3d9dae0550b3d348da811d4dde5adb4a2f77514faf8f73717645836595b7"} Sep 18 17:29:44.831463 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.831382 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb" event={"ID":"79bea44d-81c3-44bd-b905-fea0e2181f50","Type":"ContainerDied","Data":"892c0178efb851577624723b9fff11d606fdceb11da707f32062141358856fc9"} Sep 18 17:29:44.831463 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.831406 2560 scope.go:117] "RemoveContainer" containerID="828d3d9dae0550b3d348da811d4dde5adb4a2f77514faf8f73717645836595b7" Sep 18 17:29:44.841234 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.841216 2560 scope.go:117] "RemoveContainer" containerID="e7a6e2f2bb5f58e4f258285cb7b2d428d30f6b77e193ef7a734f3ef49635004c" Sep 18 17:29:44.851202 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.851180 2560 scope.go:117] "RemoveContainer" containerID="828d3d9dae0550b3d348da811d4dde5adb4a2f77514faf8f73717645836595b7" Sep 18 17:29:44.851597 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:29:44.851575 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828d3d9dae0550b3d348da811d4dde5adb4a2f77514faf8f73717645836595b7\": container with ID starting with 828d3d9dae0550b3d348da811d4dde5adb4a2f77514faf8f73717645836595b7 not found: ID does not exist" containerID="828d3d9dae0550b3d348da811d4dde5adb4a2f77514faf8f73717645836595b7" Sep 18 17:29:44.851676 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.851605 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828d3d9dae0550b3d348da811d4dde5adb4a2f77514faf8f73717645836595b7"} err="failed to get container status \"828d3d9dae0550b3d348da811d4dde5adb4a2f77514faf8f73717645836595b7\": rpc error: code = NotFound desc = could not find container \"828d3d9dae0550b3d348da811d4dde5adb4a2f77514faf8f73717645836595b7\": container with ID starting with 828d3d9dae0550b3d348da811d4dde5adb4a2f77514faf8f73717645836595b7 not found: ID does not exist" Sep 18 17:29:44.851676 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.851620 2560 scope.go:117] "RemoveContainer" containerID="e7a6e2f2bb5f58e4f258285cb7b2d428d30f6b77e193ef7a734f3ef49635004c" Sep 18 17:29:44.851903 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:29:44.851882 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a6e2f2bb5f58e4f258285cb7b2d428d30f6b77e193ef7a734f3ef49635004c\": container with ID starting with e7a6e2f2bb5f58e4f258285cb7b2d428d30f6b77e193ef7a734f3ef49635004c not found: ID does not exist" containerID="e7a6e2f2bb5f58e4f258285cb7b2d428d30f6b77e193ef7a734f3ef49635004c" Sep 18 17:29:44.851952 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.851908 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a6e2f2bb5f58e4f258285cb7b2d428d30f6b77e193ef7a734f3ef49635004c"} err="failed to get container status \"e7a6e2f2bb5f58e4f258285cb7b2d428d30f6b77e193ef7a734f3ef49635004c\": rpc error: code = NotFound desc = could not find container \"e7a6e2f2bb5f58e4f258285cb7b2d428d30f6b77e193ef7a734f3ef49635004c\": container with ID starting with e7a6e2f2bb5f58e4f258285cb7b2d428d30f6b77e193ef7a734f3ef49635004c not found: ID does not exist" Sep 18 17:29:44.855469 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.855447 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb"] Sep 18 17:29:44.866215 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:44.866195 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-fb1cf26a-openshift-default-5888c6b544-6qgsb"] Sep 18 17:29:46.758469 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:46.758439 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:46.865076 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:46.865043 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79bea44d-81c3-44bd-b905-fea0e2181f50" path="/var/lib/kubelet/pods/79bea44d-81c3-44bd-b905-fea0e2181f50/volumes" Sep 18 17:29:47.786107 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:47.786078 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:47.945523 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:47.945468 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:48.061036 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:48.060939 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:48.149667 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:48.149618 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:48.520350 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:48.520317 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:29:48.861059 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:48.860959 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:29:48.861649 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:48.861627 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:29:48.868110 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:48.868082 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:29:48.868515 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:29:48.868495 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:30:00.160596 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.160562 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk"] Sep 18 17:30:00.161066 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.161053 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33" containerName="maas-api" Sep 18 17:30:00.161066 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.161067 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33" containerName="maas-api" Sep 18 17:30:00.161139 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.161089 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bea44d-81c3-44bd-b905-fea0e2181f50" containerName="istio-proxy" Sep 18 17:30:00.161139 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.161095 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bea44d-81c3-44bd-b905-fea0e2181f50" containerName="istio-proxy" Sep 18 17:30:00.161139 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.161106 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bea44d-81c3-44bd-b905-fea0e2181f50" containerName="istio-proxy" Sep 18 17:30:00.161139 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.161111 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bea44d-81c3-44bd-b905-fea0e2181f50" containerName="istio-proxy" Sep 18 17:30:00.161258 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.161190 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="79bea44d-81c3-44bd-b905-fea0e2181f50" containerName="istio-proxy" Sep 18 17:30:00.161258 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.161204 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c9fe9d3-27ac-4c01-bbb5-7d1373fd4e33" containerName="maas-api" Sep 18 17:30:00.161258 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.161210 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="79bea44d-81c3-44bd-b905-fea0e2181f50" containerName="istio-proxy" Sep 18 17:30:00.164320 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.164274 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk" Sep 18 17:30:00.167549 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.167524 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\"" Sep 18 17:30:00.177839 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.177370 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk" Sep 18 17:30:00.177839 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.177631 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk"] Sep 18 17:30:00.451316 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.450104 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk"] Sep 18 17:30:00.923633 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.923601 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk" event={"ID":"f9cf2806-1599-4e12-9c0f-4e82a5a029ef","Type":"ContainerStarted","Data":"b3b83cf985b8a24204dd3097a231ca54cb433486100888fb3b79590464cc770f"} Sep 18 17:30:00.923813 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.923646 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk" event={"ID":"f9cf2806-1599-4e12-9c0f-4e82a5a029ef","Type":"ContainerStarted","Data":"c25d7b9295b9207181abee10c961746922d0a86ab50b8e019df364629ca99b7d"} Sep 18 17:30:00.960182 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:00.960138 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk" podStartSLOduration=0.960123911 podStartE2EDuration="960.123911ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:00.959186173 +0000 UTC m=+914.212346989" watchObservedRunningTime="2026-09-18 17:30:00.960123911 +0000 UTC m=+914.213284726" Sep 18 17:30:01.717900 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:01.717821 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:30:01.930901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:01.930872 2560 generic.go:356] "Generic (PLEG): container finished" podID="f9cf2806-1599-4e12-9c0f-4e82a5a029ef" containerID="b3b83cf985b8a24204dd3097a231ca54cb433486100888fb3b79590464cc770f" exitCode=0 Sep 18 17:30:01.931074 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:01.930927 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk" event={"ID":"f9cf2806-1599-4e12-9c0f-4e82a5a029ef","Type":"ContainerDied","Data":"b3b83cf985b8a24204dd3097a231ca54cb433486100888fb3b79590464cc770f"} Sep 18 17:30:02.963269 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:02.963220 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:03.086095 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:03.086072 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk" Sep 18 17:30:03.944078 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:03.944045 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk" Sep 18 17:30:03.944257 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:03.944098 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk" event={"ID":"f9cf2806-1599-4e12-9c0f-4e82a5a029ef","Type":"ContainerDied","Data":"c25d7b9295b9207181abee10c961746922d0a86ab50b8e019df364629ca99b7d"} Sep 18 17:30:03.944257 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:03.944155 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c25d7b9295b9207181abee10c961746922d0a86ab50b8e019df364629ca99b7d" Sep 18 17:30:04.952075 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:04.952012 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:05.034039 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:05.034010 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:14.457511 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:14.457477 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:16.016048 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:16.016018 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:16.263591 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:16.263557 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv"] Sep 18 17:30:16.263809 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:16.263784 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" containerName="istio-proxy" containerID="cri-o://99eb3c25c9933dec8ef83d9d5007a1a94a38f4caf331160d6bfa04855b1c8005" gracePeriod=30 Sep 18 17:30:16.330260 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:16.330181 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:16.728673 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:16.728642 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:16.825635 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:16.825608 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:16.993549 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:16.993458 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp"] Sep 18 17:30:16.993982 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:16.993964 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9cf2806-1599-4e12-9c0f-4e82a5a029ef" containerName="cleanup" Sep 18 17:30:16.993982 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:16.993980 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9cf2806-1599-4e12-9c0f-4e82a5a029ef" containerName="cleanup" Sep 18 17:30:16.994185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:16.994077 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9cf2806-1599-4e12-9c0f-4e82a5a029ef" containerName="cleanup" Sep 18 17:30:16.996547 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:16.996532 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.001062 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.001042 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-97798a-openshift-default-dockercfg-bm68t\"" Sep 18 17:30:17.018725 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.018697 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp"] Sep 18 17:30:17.052780 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.051941 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:17.091441 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.091404 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-workload-socket\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.091663 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.091455 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-data\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.091663 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.091601 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-credential-socket\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.091797 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.091683 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-workload-certs\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.091797 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.091714 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-podinfo\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.091905 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.091799 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv2pp\" (UniqueName: \"kubernetes.io/projected/488ffbe5-4dff-4462-bdde-4b207a3fa156-kube-api-access-sv2pp\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.091905 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.091848 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-envoy\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.091905 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.091881 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-token\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.092058 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.091905 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488ffbe5-4dff-4462-bdde-4b207a3fa156-istiod-ca-cert\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.192788 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.192748 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-credential-socket\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.192991 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.192804 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-workload-certs\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.192991 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.192822 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-podinfo\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.192991 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.192854 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv2pp\" (UniqueName: \"kubernetes.io/projected/488ffbe5-4dff-4462-bdde-4b207a3fa156-kube-api-access-sv2pp\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.192991 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.192882 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-envoy\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.192991 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.192903 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-token\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.193274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.193072 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488ffbe5-4dff-4462-bdde-4b207a3fa156-istiod-ca-cert\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.193274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.193185 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-credential-socket\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.193274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.193211 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-workload-socket\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.193274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.193239 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-data\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.193570 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.193544 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-workload-socket\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.193695 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.193575 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-data\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.193695 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.193678 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-workload-certs\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.193817 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.193798 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488ffbe5-4dff-4462-bdde-4b207a3fa156-istiod-ca-cert\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.195354 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.195332 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-envoy\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.195618 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.195596 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-podinfo\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.230025 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.229994 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-token\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.230972 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.230939 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv2pp\" (UniqueName: \"kubernetes.io/projected/488ffbe5-4dff-4462-bdde-4b207a3fa156-kube-api-access-sv2pp\") pod \"e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.242591 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.242563 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:17.306723 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.306632 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:17.610655 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:30:17.610623 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488ffbe5_4dff_4462_bdde_4b207a3fa156.slice/crio-7807206c35fba181b473532ebdf42b0cc00c2b23b38bbe154ba10ea6f298b6ae WatchSource:0}: Error finding container 7807206c35fba181b473532ebdf42b0cc00c2b23b38bbe154ba10ea6f298b6ae: Status 404 returned error can't find the container with id 7807206c35fba181b473532ebdf42b0cc00c2b23b38bbe154ba10ea6f298b6ae Sep 18 17:30:17.611545 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.611524 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp"] Sep 18 17:30:17.612950 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.612920 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:30:17.612999 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.612983 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:30:17.613048 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:17.613011 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:30:18.071953 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:18.071921 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" event={"ID":"488ffbe5-4dff-4462-bdde-4b207a3fa156","Type":"ContainerStarted","Data":"3a46d40bec19cf492166764fcf526b137cd3eeed72461cc045408551ef5b89bd"} Sep 18 17:30:18.072354 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:18.071972 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" event={"ID":"488ffbe5-4dff-4462-bdde-4b207a3fa156","Type":"ContainerStarted","Data":"7807206c35fba181b473532ebdf42b0cc00c2b23b38bbe154ba10ea6f298b6ae"} Sep 18 17:30:18.096202 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:18.096155 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" podStartSLOduration=2.096140999 podStartE2EDuration="2.096140999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:18.095310348 +0000 UTC m=+931.348471174" watchObservedRunningTime="2026-09-18 17:30:18.096140999 +0000 UTC m=+931.349301814" Sep 18 17:30:18.306878 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:18.306837 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:18.311982 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:18.311956 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:19.077347 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:19.077314 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:19.078368 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:19.078346 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:19.451327 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:19.451279 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:20.061914 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:20.061886 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:20.157320 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:20.157269 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:20.625655 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:20.625628 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:21.438815 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.438777 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:30:21.512014 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.511991 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:30:21.640748 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.640661 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-istio-data\") pod \"c57fd2f9-3bf7-4dee-8997-483002041249\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " Sep 18 17:30:21.640916 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.640758 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-credential-socket\") pod \"c57fd2f9-3bf7-4dee-8997-483002041249\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " Sep 18 17:30:21.640916 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.640780 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57fd2f9-3bf7-4dee-8997-483002041249-istio-token\") pod \"c57fd2f9-3bf7-4dee-8997-483002041249\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " Sep 18 17:30:21.640916 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.640812 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57fd2f9-3bf7-4dee-8997-483002041249-istio-podinfo\") pod \"c57fd2f9-3bf7-4dee-8997-483002041249\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " Sep 18 17:30:21.640916 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.640840 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-workload-socket\") pod \"c57fd2f9-3bf7-4dee-8997-483002041249\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " Sep 18 17:30:21.640916 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.640873 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-workload-certs\") pod \"c57fd2f9-3bf7-4dee-8997-483002041249\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " Sep 18 17:30:21.641174 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.640963 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57fd2f9-3bf7-4dee-8997-483002041249-istiod-ca-cert\") pod \"c57fd2f9-3bf7-4dee-8997-483002041249\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " Sep 18 17:30:21.641174 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.640980 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-istio-data" (OuterVolumeSpecName: "istio-data") pod "c57fd2f9-3bf7-4dee-8997-483002041249" (UID: "c57fd2f9-3bf7-4dee-8997-483002041249"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:30:21.641174 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.641068 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-istio-envoy\") pod \"c57fd2f9-3bf7-4dee-8997-483002041249\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " Sep 18 17:30:21.641174 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.641079 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c57fd2f9-3bf7-4dee-8997-483002041249" (UID: "c57fd2f9-3bf7-4dee-8997-483002041249"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:30:21.641174 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.641109 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmmz6\" (UniqueName: \"kubernetes.io/projected/c57fd2f9-3bf7-4dee-8997-483002041249-kube-api-access-nmmz6\") pod \"c57fd2f9-3bf7-4dee-8997-483002041249\" (UID: \"c57fd2f9-3bf7-4dee-8997-483002041249\") " Sep 18 17:30:21.641174 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.641104 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c57fd2f9-3bf7-4dee-8997-483002041249" (UID: "c57fd2f9-3bf7-4dee-8997-483002041249"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:30:21.641174 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.641165 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c57fd2f9-3bf7-4dee-8997-483002041249" (UID: "c57fd2f9-3bf7-4dee-8997-483002041249"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:30:21.641582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.641379 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57fd2f9-3bf7-4dee-8997-483002041249-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c57fd2f9-3bf7-4dee-8997-483002041249" (UID: "c57fd2f9-3bf7-4dee-8997-483002041249"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:30:21.641582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.641563 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:30:21.641674 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.641583 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:30:21.641674 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.641603 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:30:21.641674 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.641620 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57fd2f9-3bf7-4dee-8997-483002041249-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:30:21.641674 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.641635 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:30:21.643148 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.643118 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c57fd2f9-3bf7-4dee-8997-483002041249" (UID: "c57fd2f9-3bf7-4dee-8997-483002041249"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:30:21.643267 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.643244 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c57fd2f9-3bf7-4dee-8997-483002041249-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c57fd2f9-3bf7-4dee-8997-483002041249" (UID: "c57fd2f9-3bf7-4dee-8997-483002041249"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:30:21.643505 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.643467 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57fd2f9-3bf7-4dee-8997-483002041249-istio-token" (OuterVolumeSpecName: "istio-token") pod "c57fd2f9-3bf7-4dee-8997-483002041249" (UID: "c57fd2f9-3bf7-4dee-8997-483002041249"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:30:21.643634 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.643504 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57fd2f9-3bf7-4dee-8997-483002041249-kube-api-access-nmmz6" (OuterVolumeSpecName: "kube-api-access-nmmz6") pod "c57fd2f9-3bf7-4dee-8997-483002041249" (UID: "c57fd2f9-3bf7-4dee-8997-483002041249"). InnerVolumeSpecName "kube-api-access-nmmz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:30:21.742718 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.742683 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57fd2f9-3bf7-4dee-8997-483002041249-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:30:21.742718 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.742713 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57fd2f9-3bf7-4dee-8997-483002041249-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:30:21.742718 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.742723 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57fd2f9-3bf7-4dee-8997-483002041249-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:30:21.742948 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.742731 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmmz6\" (UniqueName: \"kubernetes.io/projected/c57fd2f9-3bf7-4dee-8997-483002041249-kube-api-access-nmmz6\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:30:21.769786 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:21.769757 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:22.096187 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.096155 2560 generic.go:356] "Generic (PLEG): container finished" podID="c57fd2f9-3bf7-4dee-8997-483002041249" containerID="99eb3c25c9933dec8ef83d9d5007a1a94a38f4caf331160d6bfa04855b1c8005" exitCode=0 Sep 18 17:30:22.096389 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.096205 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" Sep 18 17:30:22.096389 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.096235 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" event={"ID":"c57fd2f9-3bf7-4dee-8997-483002041249","Type":"ContainerDied","Data":"99eb3c25c9933dec8ef83d9d5007a1a94a38f4caf331160d6bfa04855b1c8005"} Sep 18 17:30:22.096389 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.096306 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv" event={"ID":"c57fd2f9-3bf7-4dee-8997-483002041249","Type":"ContainerDied","Data":"b73a300ef932aca00d5880294406fffacedf60f45a8d4c4371901e191385e6aa"} Sep 18 17:30:22.096389 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.096331 2560 scope.go:117] "RemoveContainer" containerID="99eb3c25c9933dec8ef83d9d5007a1a94a38f4caf331160d6bfa04855b1c8005" Sep 18 17:30:22.111174 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.111141 2560 scope.go:117] "RemoveContainer" containerID="7881323fd83b464db168f5a41b0718a2d0568e1a6528e67ed238d00169b01446" Sep 18 17:30:22.120503 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.120480 2560 scope.go:117] "RemoveContainer" containerID="99eb3c25c9933dec8ef83d9d5007a1a94a38f4caf331160d6bfa04855b1c8005" Sep 18 17:30:22.120798 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:30:22.120771 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99eb3c25c9933dec8ef83d9d5007a1a94a38f4caf331160d6bfa04855b1c8005\": container with ID starting with 99eb3c25c9933dec8ef83d9d5007a1a94a38f4caf331160d6bfa04855b1c8005 not found: ID does not exist" containerID="99eb3c25c9933dec8ef83d9d5007a1a94a38f4caf331160d6bfa04855b1c8005" Sep 18 17:30:22.120872 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.120796 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99eb3c25c9933dec8ef83d9d5007a1a94a38f4caf331160d6bfa04855b1c8005"} err="failed to get container status \"99eb3c25c9933dec8ef83d9d5007a1a94a38f4caf331160d6bfa04855b1c8005\": rpc error: code = NotFound desc = could not find container \"99eb3c25c9933dec8ef83d9d5007a1a94a38f4caf331160d6bfa04855b1c8005\": container with ID starting with 99eb3c25c9933dec8ef83d9d5007a1a94a38f4caf331160d6bfa04855b1c8005 not found: ID does not exist" Sep 18 17:30:22.120872 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.120816 2560 scope.go:117] "RemoveContainer" containerID="7881323fd83b464db168f5a41b0718a2d0568e1a6528e67ed238d00169b01446" Sep 18 17:30:22.121048 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:30:22.121029 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7881323fd83b464db168f5a41b0718a2d0568e1a6528e67ed238d00169b01446\": container with ID starting with 7881323fd83b464db168f5a41b0718a2d0568e1a6528e67ed238d00169b01446 not found: ID does not exist" containerID="7881323fd83b464db168f5a41b0718a2d0568e1a6528e67ed238d00169b01446" Sep 18 17:30:22.121088 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.121051 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7881323fd83b464db168f5a41b0718a2d0568e1a6528e67ed238d00169b01446"} err="failed to get container status \"7881323fd83b464db168f5a41b0718a2d0568e1a6528e67ed238d00169b01446\": rpc error: code = NotFound desc = could not find container \"7881323fd83b464db168f5a41b0718a2d0568e1a6528e67ed238d00169b01446\": container with ID starting with 7881323fd83b464db168f5a41b0718a2d0568e1a6528e67ed238d00169b01446 not found: ID does not exist" Sep 18 17:30:22.131855 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.131831 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv"] Sep 18 17:30:22.137626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.137604 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-535fd3-openshift-default-75fd599fd6-98zxv"] Sep 18 17:30:22.155227 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:30:22.155198 2560 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57fd2f9_3bf7_4dee_8997_483002041249.slice/crio-b73a300ef932aca00d5880294406fffacedf60f45a8d4c4371901e191385e6aa\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57fd2f9_3bf7_4dee_8997_483002041249.slice\": RecentStats: unable to find data in memory cache]" Sep 18 17:30:22.865209 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.865177 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" path="/var/lib/kubelet/pods/c57fd2f9-3bf7-4dee-8997-483002041249/volumes" Sep 18 17:30:22.938871 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:22.938841 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:23.294875 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:23.294844 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:23.400316 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:23.400269 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:23.498043 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:23.498009 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:23.856117 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:23.856077 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:25.161705 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:25.161677 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:25.241519 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:25.241489 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:25.545074 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:25.544971 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:25.739136 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:25.739107 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:25.826257 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:25.826182 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:26.015487 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:26.015450 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:39.356854 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:39.356816 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:41.185401 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:41.185374 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:46.209280 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:46.209245 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:46.249628 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:46.249593 2560 generic.go:356] "Generic (PLEG): container finished" podID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerID="3a46d40bec19cf492166764fcf526b137cd3eeed72461cc045408551ef5b89bd" exitCode=0 Sep 18 17:30:46.249794 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:46.249662 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" event={"ID":"488ffbe5-4dff-4462-bdde-4b207a3fa156","Type":"ContainerDied","Data":"3a46d40bec19cf492166764fcf526b137cd3eeed72461cc045408551ef5b89bd"} Sep 18 17:30:46.250078 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:46.250066 2560 scope.go:117] "RemoveContainer" containerID="3a46d40bec19cf492166764fcf526b137cd3eeed72461cc045408551ef5b89bd" Sep 18 17:30:46.251360 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:46.251334 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:30:46.251426 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:46.251398 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:30:46.251479 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:46.251426 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:30:46.290062 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:46.290040 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:46.368739 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:46.368704 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:46.466172 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:46.466080 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:47.261615 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:47.261575 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" event={"ID":"488ffbe5-4dff-4462-bdde-4b207a3fa156","Type":"ContainerStarted","Data":"711593217bcc3ae00aa17e5c77ad0f5480474896a32ac7bfc3434911fd801097"} Sep 18 17:30:47.307478 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:47.307448 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:47.308515 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:47.308490 2560 patch_prober.go:28] interesting pod/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 18 17:30:47.308628 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:47.308537 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 18 17:30:47.405883 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:47.405848 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:30:48.274348 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:48.274315 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:48.307701 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:48.307662 2560 patch_prober.go:28] interesting pod/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 18 17:30:48.307896 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:48.307726 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 18 17:30:49.077566 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:49.077527 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:49.307711 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:49.307632 2560 patch_prober.go:28] interesting pod/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 18 17:30:49.307711 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:49.307693 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 18 17:30:49.389413 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:49.389349 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:50.129900 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:50.129866 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:50.307669 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:50.307628 2560 patch_prober.go:28] interesting pod/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 18 17:30:50.307857 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:50.307702 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 18 17:30:50.614650 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:50.614617 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:51.306965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:51.306928 2560 patch_prober.go:28] interesting pod/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 18 17:30:51.307179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:51.306995 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 18 17:30:51.719284 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:51.719253 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:52.307265 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:52.307220 2560 patch_prober.go:28] interesting pod/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 18 17:30:52.307487 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:52.307287 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 18 17:30:52.764384 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:52.764357 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:53.311599 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:53.311572 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:53.312532 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:53.312509 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:30:53.829537 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:53.829504 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:56.910441 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:56.910392 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:30:57.681699 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:30:57.681665 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:04.100260 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:04.100180 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:04.182963 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:04.182929 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:04.271753 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:04.271721 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:04.363449 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:04.363361 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:05.032223 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:05.032184 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:05.145766 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:05.145729 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:05.241985 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:05.241953 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:05.818897 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:05.818869 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:31:06.677248 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:06.677218 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:12.938320 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:12.938266 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:13.005892 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:13.005864 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:13.652468 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:13.652435 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:20.247068 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.247034 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:20.305148 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.305119 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:20.482137 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.482102 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:20.773481 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.773448 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv"] Sep 18 17:31:20.773947 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.773935 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" containerName="istio-proxy" Sep 18 17:31:20.774005 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.773950 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" containerName="istio-proxy" Sep 18 17:31:20.774043 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.774029 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" containerName="istio-proxy" Sep 18 17:31:20.774043 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.774039 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" containerName="istio-proxy" Sep 18 17:31:20.776364 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.776343 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.779339 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.779309 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-55a2c1b6-openshift-default-dockercfg-bfw6b\"" Sep 18 17:31:20.791244 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.791216 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv"] Sep 18 17:31:20.796345 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.796307 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:20.868329 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.866100 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87251652-60d8-4833-a3ab-aab4d31e8063-istio-podinfo\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.868329 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.866335 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87251652-60d8-4833-a3ab-aab4d31e8063-istiod-ca-cert\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.868329 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.866373 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzhn\" (UniqueName: \"kubernetes.io/projected/87251652-60d8-4833-a3ab-aab4d31e8063-kube-api-access-nlzhn\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.868329 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.866442 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87251652-60d8-4833-a3ab-aab4d31e8063-istio-token\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.868329 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.866478 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-istio-data\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.868329 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.866507 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-istio-envoy\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.868329 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.866620 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-workload-socket\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.868329 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.866648 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-credential-socket\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.868329 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.866675 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-workload-certs\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.968085 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.968050 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-workload-socket\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.968085 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.968088 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-credential-socket\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.968375 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.968109 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-workload-certs\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.968375 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.968260 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87251652-60d8-4833-a3ab-aab4d31e8063-istio-podinfo\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.968490 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.968437 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87251652-60d8-4833-a3ab-aab4d31e8063-istiod-ca-cert\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.968490 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.968475 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzhn\" (UniqueName: \"kubernetes.io/projected/87251652-60d8-4833-a3ab-aab4d31e8063-kube-api-access-nlzhn\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.968600 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.968560 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87251652-60d8-4833-a3ab-aab4d31e8063-istio-token\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.968655 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.968606 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-istio-data\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.968655 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.968637 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-istio-envoy\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.968762 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.968716 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-workload-socket\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.968762 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.968565 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-credential-socket\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.969026 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.968992 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-workload-certs\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.969174 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.969094 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-istio-data\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.969243 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.969201 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87251652-60d8-4833-a3ab-aab4d31e8063-istiod-ca-cert\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.970973 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.970949 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-istio-envoy\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.971110 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.971012 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87251652-60d8-4833-a3ab-aab4d31e8063-istio-podinfo\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.977130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.977108 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87251652-60d8-4833-a3ab-aab4d31e8063-istio-token\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:20.977414 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:20.977391 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzhn\" (UniqueName: \"kubernetes.io/projected/87251652-60d8-4833-a3ab-aab4d31e8063-kube-api-access-nlzhn\") pod \"e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:21.088079 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:21.087974 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:21.106905 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:21.106844 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:21.433687 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:31:21.433648 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87251652_60d8_4833_a3ab_aab4d31e8063.slice/crio-4cd98eaf8e4f41822c0c4e23bd9a0b00f20971cc367abf5b122431c7c2c4e4d2 WatchSource:0}: Error finding container 4cd98eaf8e4f41822c0c4e23bd9a0b00f20971cc367abf5b122431c7c2c4e4d2: Status 404 returned error can't find the container with id 4cd98eaf8e4f41822c0c4e23bd9a0b00f20971cc367abf5b122431c7c2c4e4d2 Sep 18 17:31:21.436259 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:21.436223 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:31:21.436373 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:21.436315 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:31:21.436373 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:21.436350 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:31:21.437625 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:21.437601 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv"] Sep 18 17:31:21.473871 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:21.473839 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" event={"ID":"87251652-60d8-4833-a3ab-aab4d31e8063","Type":"ContainerStarted","Data":"4cd98eaf8e4f41822c0c4e23bd9a0b00f20971cc367abf5b122431c7c2c4e4d2"} Sep 18 17:31:22.481507 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:22.481472 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" event={"ID":"87251652-60d8-4833-a3ab-aab4d31e8063","Type":"ContainerStarted","Data":"c9b9be90f9abd24f3eac086660f45ab8bcfcf1f890dc0598c9dff6b3cad7d984"} Sep 18 17:31:22.504465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:22.504409 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" podStartSLOduration=2.504393906 podStartE2EDuration="2.504393906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:31:22.50241983 +0000 UTC m=+995.755580672" watchObservedRunningTime="2026-09-18 17:31:22.504393906 +0000 UTC m=+995.757554723" Sep 18 17:31:23.089079 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:23.089041 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:23.093845 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:23.093823 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:23.487060 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:23.487030 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:23.488048 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:23.488029 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:26.523710 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:26.523649 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:27.204120 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:27.204091 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:27.301499 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:27.301464 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:27.625623 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:27.625518 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:27.909729 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:27.909697 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:31:29.005537 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:29.005498 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:29.954763 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:29.954733 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:30.906957 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:30.906926 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:31.718095 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:31.718064 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:32.414240 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.414196 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt"] Sep 18 17:31:32.414811 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.414794 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" containerName="istio-proxy" Sep 18 17:31:32.414811 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.414812 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57fd2f9-3bf7-4dee-8997-483002041249" containerName="istio-proxy" Sep 18 17:31:32.417417 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.417399 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.420013 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.419990 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-55a2c1b6-serving-cert\"" Sep 18 17:31:32.433652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.433619 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt"] Sep 18 17:31:32.447317 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.443811 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:32.525774 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.525729 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:32.597020 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.596980 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfec1d3-56bd-4f4c-9377-5526d3124459-openshift-service-ca\") pod \"maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.597204 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.597046 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-924qk\" (UniqueName: \"kubernetes.io/projected/bbfec1d3-56bd-4f4c-9377-5526d3124459-kube-api-access-924qk\") pod \"maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.597204 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.597122 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfec1d3-56bd-4f4c-9377-5526d3124459-metrics-certs\") pod \"maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.597204 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.597202 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfec1d3-56bd-4f4c-9377-5526d3124459-maas-api-tls\") pod \"maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.693557 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.693441 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:32.698460 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.698437 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfec1d3-56bd-4f4c-9377-5526d3124459-openshift-service-ca\") pod \"maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.698551 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.698480 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-924qk\" (UniqueName: \"kubernetes.io/projected/bbfec1d3-56bd-4f4c-9377-5526d3124459-kube-api-access-924qk\") pod \"maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.698551 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.698517 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfec1d3-56bd-4f4c-9377-5526d3124459-metrics-certs\") pod \"maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.698661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.698568 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfec1d3-56bd-4f4c-9377-5526d3124459-maas-api-tls\") pod \"maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.699062 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.699043 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfec1d3-56bd-4f4c-9377-5526d3124459-openshift-service-ca\") pod \"maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.701005 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.700985 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfec1d3-56bd-4f4c-9377-5526d3124459-maas-api-tls\") pod \"maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.701106 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.701038 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfec1d3-56bd-4f4c-9377-5526d3124459-metrics-certs\") pod \"maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.706701 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.706673 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-924qk\" (UniqueName: \"kubernetes.io/projected/bbfec1d3-56bd-4f4c-9377-5526d3124459-kube-api-access-924qk\") pod \"maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.732156 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.732133 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:32.916190 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:32.916157 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:33.008009 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:33.007984 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt"] Sep 18 17:31:33.009439 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:31:33.009410 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfec1d3_56bd_4f4c_9377_5526d3124459.slice/crio-e6ce6c2b727e9552fd0d01d4e6d1b8467892c2dc3e346bb936bbd77e89a9f843 WatchSource:0}: Error finding container e6ce6c2b727e9552fd0d01d4e6d1b8467892c2dc3e346bb936bbd77e89a9f843: Status 404 returned error can't find the container with id e6ce6c2b727e9552fd0d01d4e6d1b8467892c2dc3e346bb936bbd77e89a9f843 Sep 18 17:31:33.010255 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:33.010226 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:33.547765 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:33.547737 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" event={"ID":"bbfec1d3-56bd-4f4c-9377-5526d3124459","Type":"ContainerStarted","Data":"e6ce6c2b727e9552fd0d01d4e6d1b8467892c2dc3e346bb936bbd77e89a9f843"} Sep 18 17:31:34.558640 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:34.558591 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" event={"ID":"bbfec1d3-56bd-4f4c-9377-5526d3124459","Type":"ContainerStarted","Data":"ea2adc40d9431397a7b1d2c2559f9048595a70878f387779fceb86d83adfa8c2"} Sep 18 17:31:34.559047 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:34.558822 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:34.579662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:34.579602 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" podStartSLOduration=2.06213795 podStartE2EDuration="2.579584568s" totalImagesPullingTime="517.446618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:31:34.578453286 +0000 UTC m=+1007.831614124" watchObservedRunningTime="2026-09-18 17:31:34.579584568 +0000 UTC m=+1007.832745444" Sep 18 17:31:40.570539 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:40.570511 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:31:43.254221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:43.254182 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:47.714548 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:47.714512 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:47.807159 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:47.807129 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:54.528502 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:54.528466 2560 patch_prober.go:28] interesting pod/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 18 17:31:54.528919 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:54.528549 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 18 17:31:54.541579 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:54.541546 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:55.219769 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:55.219728 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:56.024894 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:56.024863 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:56.717821 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:56.717787 2560 generic.go:356] "Generic (PLEG): container finished" podID="87251652-60d8-4833-a3ab-aab4d31e8063" containerID="c9b9be90f9abd24f3eac086660f45ab8bcfcf1f890dc0598c9dff6b3cad7d984" exitCode=0 Sep 18 17:31:56.718010 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:56.717846 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" event={"ID":"87251652-60d8-4833-a3ab-aab4d31e8063","Type":"ContainerDied","Data":"c9b9be90f9abd24f3eac086660f45ab8bcfcf1f890dc0598c9dff6b3cad7d984"} Sep 18 17:31:56.718312 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:56.718285 2560 scope.go:117] "RemoveContainer" containerID="c9b9be90f9abd24f3eac086660f45ab8bcfcf1f890dc0598c9dff6b3cad7d984" Sep 18 17:31:56.719482 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:56.719449 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:31:56.719548 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:56.719535 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:31:56.719590 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:56.719571 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:31:56.950935 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:56.950892 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:57.262167 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:57.262123 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:57.685183 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:57.685154 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:57.725401 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:57.725368 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" event={"ID":"87251652-60d8-4833-a3ab-aab4d31e8063","Type":"ContainerStarted","Data":"e8978fa03c3c2485b0f6f5808bfa17b3de644224ac912fdf480ecb55d269f644"} Sep 18 17:31:57.933445 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:57.933414 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:58.032120 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:58.032032 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:58.088950 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:58.088917 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:31:58.089141 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:58.089124 2560 patch_prober.go:28] interesting pod/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 18 17:31:58.089201 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:58.089182 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 18 17:31:58.329455 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:58.329371 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:58.639201 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:58.639085 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:31:59.088695 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:59.088662 2560 patch_prober.go:28] interesting pod/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 18 17:31:59.088895 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:59.088718 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 18 17:31:59.534488 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:31:59.534453 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:00.089242 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:00.089209 2560 patch_prober.go:28] interesting pod/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 18 17:32:00.089422 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:00.089263 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 18 17:32:01.089334 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:01.089282 2560 patch_prober.go:28] interesting pod/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 18 17:32:01.089728 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:01.089357 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 18 17:32:02.088685 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:02.088647 2560 patch_prober.go:28] interesting pod/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 18 17:32:02.088939 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:02.088705 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 18 17:32:03.092901 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:03.092867 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:32:03.093333 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:03.093259 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:32:03.094016 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:03.093994 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:32:03.747795 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:03.747763 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:04.218724 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:04.218689 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:04.731509 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:04.731480 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:04.872604 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:04.872574 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:04.905397 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:04.905363 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:32:05.060637 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:05.060522 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:06.064874 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:06.064843 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:06.823460 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:06.823429 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:07.632302 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:07.632256 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:08.324510 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:08.324474 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:08.993873 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:08.993839 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:09.849327 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:09.849282 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:10.217094 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:10.217064 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:10.649536 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:10.649506 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:14.123236 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:14.123196 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:14.688842 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:14.688808 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:14.757852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:14.757808 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:14.856032 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:14.856001 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:14.957829 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:14.957729 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:15.225599 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:15.225516 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:15.338934 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:15.338903 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:16.107529 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:16.107495 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:17.037412 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:17.037375 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:17.699026 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:17.698986 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt"] Sep 18 17:32:17.699308 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:17.699243 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" podUID="bbfec1d3-56bd-4f4c-9377-5526d3124459" containerName="maas-api" containerID="cri-o://ea2adc40d9431397a7b1d2c2559f9048595a70878f387779fceb86d83adfa8c2" gracePeriod=30 Sep 18 17:32:17.754823 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:17.754560 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:17.857466 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:17.857428 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:17.858666 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:17.858643 2560 generic.go:356] "Generic (PLEG): container finished" podID="bbfec1d3-56bd-4f4c-9377-5526d3124459" containerID="ea2adc40d9431397a7b1d2c2559f9048595a70878f387779fceb86d83adfa8c2" exitCode=0 Sep 18 17:32:17.858787 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:17.858676 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" event={"ID":"bbfec1d3-56bd-4f4c-9377-5526d3124459","Type":"ContainerDied","Data":"ea2adc40d9431397a7b1d2c2559f9048595a70878f387779fceb86d83adfa8c2"} Sep 18 17:32:17.963265 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:17.963198 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:32:17.980370 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:17.980341 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:18.055778 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.055745 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:18.115987 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.110580 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfec1d3-56bd-4f4c-9377-5526d3124459-maas-api-tls\") pod \"bbfec1d3-56bd-4f4c-9377-5526d3124459\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " Sep 18 17:32:18.115987 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.110642 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfec1d3-56bd-4f4c-9377-5526d3124459-openshift-service-ca\") pod \"bbfec1d3-56bd-4f4c-9377-5526d3124459\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " Sep 18 17:32:18.115987 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.110686 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-924qk\" (UniqueName: \"kubernetes.io/projected/bbfec1d3-56bd-4f4c-9377-5526d3124459-kube-api-access-924qk\") pod \"bbfec1d3-56bd-4f4c-9377-5526d3124459\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " Sep 18 17:32:18.115987 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.110766 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfec1d3-56bd-4f4c-9377-5526d3124459-metrics-certs\") pod \"bbfec1d3-56bd-4f4c-9377-5526d3124459\" (UID: \"bbfec1d3-56bd-4f4c-9377-5526d3124459\") " Sep 18 17:32:18.115987 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.111402 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbfec1d3-56bd-4f4c-9377-5526d3124459-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bbfec1d3-56bd-4f4c-9377-5526d3124459" (UID: "bbfec1d3-56bd-4f4c-9377-5526d3124459"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:32:18.115987 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.113911 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbfec1d3-56bd-4f4c-9377-5526d3124459-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bbfec1d3-56bd-4f4c-9377-5526d3124459" (UID: "bbfec1d3-56bd-4f4c-9377-5526d3124459"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:32:18.121940 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.116527 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfec1d3-56bd-4f4c-9377-5526d3124459-kube-api-access-924qk" (OuterVolumeSpecName: "kube-api-access-924qk") pod "bbfec1d3-56bd-4f4c-9377-5526d3124459" (UID: "bbfec1d3-56bd-4f4c-9377-5526d3124459"). InnerVolumeSpecName "kube-api-access-924qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:32:18.121940 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.116870 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbfec1d3-56bd-4f4c-9377-5526d3124459-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bbfec1d3-56bd-4f4c-9377-5526d3124459" (UID: "bbfec1d3-56bd-4f4c-9377-5526d3124459"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:32:18.212607 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.212570 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfec1d3-56bd-4f4c-9377-5526d3124459-metrics-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:18.212607 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.212602 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfec1d3-56bd-4f4c-9377-5526d3124459-maas-api-tls\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:18.212607 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.212611 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfec1d3-56bd-4f4c-9377-5526d3124459-openshift-service-ca\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:18.212607 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.212621 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-924qk\" (UniqueName: \"kubernetes.io/projected/bbfec1d3-56bd-4f4c-9377-5526d3124459-kube-api-access-924qk\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:18.353171 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.353081 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:18.866624 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.866597 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" Sep 18 17:32:18.866802 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.866643 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt" event={"ID":"bbfec1d3-56bd-4f4c-9377-5526d3124459","Type":"ContainerDied","Data":"e6ce6c2b727e9552fd0d01d4e6d1b8467892c2dc3e346bb936bbd77e89a9f843"} Sep 18 17:32:18.866802 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.866703 2560 scope.go:117] "RemoveContainer" containerID="ea2adc40d9431397a7b1d2c2559f9048595a70878f387779fceb86d83adfa8c2" Sep 18 17:32:18.893895 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.893863 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt"] Sep 18 17:32:18.902246 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:18.902199 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-55a2c1b6-764d76fdf8-6plkt"] Sep 18 17:32:19.153763 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:19.153707 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:20.487648 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:20.487601 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:20.865661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:20.865560 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbfec1d3-56bd-4f4c-9377-5526d3124459" path="/var/lib/kubelet/pods/bbfec1d3-56bd-4f4c-9377-5526d3124459/volumes" Sep 18 17:32:21.905826 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:21.905793 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:23.413737 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:23.413702 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:24.272721 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:24.272676 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:24.372717 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:24.372685 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:24.518808 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:24.518770 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp"] Sep 18 17:32:24.519328 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:24.519062 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerName="istio-proxy" containerID="cri-o://711593217bcc3ae00aa17e5c77ad0f5480474896a32ac7bfc3434911fd801097" gracePeriod=30 Sep 18 17:32:24.536575 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:24.536488 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:25.010684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.010646 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:25.282450 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.282367 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb"] Sep 18 17:32:25.282924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.282910 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbfec1d3-56bd-4f4c-9377-5526d3124459" containerName="maas-api" Sep 18 17:32:25.283001 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.282928 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbfec1d3-56bd-4f4c-9377-5526d3124459" containerName="maas-api" Sep 18 17:32:25.283044 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.283002 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbfec1d3-56bd-4f4c-9377-5526d3124459" containerName="maas-api" Sep 18 17:32:25.286242 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.286217 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.288849 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.288825 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-231bbc-openshift-default-dockercfg-26j8x\"" Sep 18 17:32:25.298140 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.298110 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb"] Sep 18 17:32:25.308570 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.308532 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:25.394875 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.394842 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-workload-socket\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.395092 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.394965 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-workload-certs\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.395092 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.395017 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-token\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.395092 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.395060 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-credential-socket\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.395260 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.395145 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-podinfo\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.395260 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.395186 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-data\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.395260 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.395239 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b9c3a5-2f82-4869-942c-30a5701716b1-istiod-ca-cert\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.395443 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.395267 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txwk\" (UniqueName: \"kubernetes.io/projected/08b9c3a5-2f82-4869-942c-30a5701716b1-kube-api-access-4txwk\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.395443 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.395321 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-envoy\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.436495 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.436462 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:25.496244 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.496200 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-workload-socket\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.496479 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.496449 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-workload-certs\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.496559 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.496503 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-token\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.496635 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.496560 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-credential-socket\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.496635 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.496597 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-podinfo\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.496755 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.496740 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-data\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.496861 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.496840 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b9c3a5-2f82-4869-942c-30a5701716b1-istiod-ca-cert\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.497106 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.497087 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txwk\" (UniqueName: \"kubernetes.io/projected/08b9c3a5-2f82-4869-942c-30a5701716b1-kube-api-access-4txwk\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.497232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.497219 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-envoy\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.497663 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.497598 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-workload-socket\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.498279 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.498259 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-data\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.499461 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.498392 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-credential-socket\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.499595 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.498554 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b9c3a5-2f82-4869-942c-30a5701716b1-istiod-ca-cert\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.499694 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.498621 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-workload-certs\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.501500 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.501451 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-envoy\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.501707 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.501685 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-podinfo\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.509979 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.509669 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txwk\" (UniqueName: \"kubernetes.io/projected/08b9c3a5-2f82-4869-942c-30a5701716b1-kube-api-access-4txwk\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.510596 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.510573 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-token\") pod \"e2e-noten-231bbc-openshift-default-86688568c4-k5wpb\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.597552 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.597456 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:25.883396 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.883366 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb"] Sep 18 17:32:25.885518 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:32:25.885474 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b9c3a5_2f82_4869_942c_30a5701716b1.slice/crio-4236a8be737315555ab896ed2059bb66f57120e529ff200a6ebbbb79dfc60f24 WatchSource:0}: Error finding container 4236a8be737315555ab896ed2059bb66f57120e529ff200a6ebbbb79dfc60f24: Status 404 returned error can't find the container with id 4236a8be737315555ab896ed2059bb66f57120e529ff200a6ebbbb79dfc60f24 Sep 18 17:32:25.887926 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.887885 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:32:25.888024 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.887981 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:32:25.888067 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.888029 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:32:25.912478 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.912448 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" event={"ID":"08b9c3a5-2f82-4869-942c-30a5701716b1","Type":"ContainerStarted","Data":"4236a8be737315555ab896ed2059bb66f57120e529ff200a6ebbbb79dfc60f24"} Sep 18 17:32:25.973069 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:25.973037 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:26.074704 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:26.074668 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:26.223769 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:26.223723 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv"] Sep 18 17:32:26.224612 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:26.224582 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" containerName="istio-proxy" containerID="cri-o://e8978fa03c3c2485b0f6f5808bfa17b3de644224ac912fdf480ecb55d269f644" gracePeriod=30 Sep 18 17:32:26.259888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:26.259830 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:26.576849 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:26.576758 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:26.673880 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:26.673840 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:26.780246 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:26.780209 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:26.921511 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:26.921476 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" event={"ID":"08b9c3a5-2f82-4869-942c-30a5701716b1","Type":"ContainerStarted","Data":"3054d640b33f6e9848451d24a7db6017b2204ca4503db1511919294824cfc167"} Sep 18 17:32:26.944732 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:26.944659 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" podStartSLOduration=1.9446403079999999 podStartE2EDuration="1.944640308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:32:26.941470089 +0000 UTC m=+1060.194630933" watchObservedRunningTime="2026-09-18 17:32:26.944640308 +0000 UTC m=+1060.197801125" Sep 18 17:32:27.239743 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:27.239652 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:27.333962 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:27.333922 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:27.436647 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:27.436610 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:27.597846 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:27.597752 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:27.603480 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:27.603451 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:27.928833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:27.928793 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:27.929730 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:27.929709 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:32:29.790549 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.790480 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:32:29.942346 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.942256 2560 generic.go:356] "Generic (PLEG): container finished" podID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerID="711593217bcc3ae00aa17e5c77ad0f5480474896a32ac7bfc3434911fd801097" exitCode=0 Sep 18 17:32:29.942346 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.942326 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" Sep 18 17:32:29.942346 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.942328 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" event={"ID":"488ffbe5-4dff-4462-bdde-4b207a3fa156","Type":"ContainerDied","Data":"711593217bcc3ae00aa17e5c77ad0f5480474896a32ac7bfc3434911fd801097"} Sep 18 17:32:29.942576 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.942378 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp" event={"ID":"488ffbe5-4dff-4462-bdde-4b207a3fa156","Type":"ContainerDied","Data":"7807206c35fba181b473532ebdf42b0cc00c2b23b38bbe154ba10ea6f298b6ae"} Sep 18 17:32:29.942576 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.942399 2560 scope.go:117] "RemoveContainer" containerID="711593217bcc3ae00aa17e5c77ad0f5480474896a32ac7bfc3434911fd801097" Sep 18 17:32:29.947642 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.947604 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-data\") pod \"488ffbe5-4dff-4462-bdde-4b207a3fa156\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " Sep 18 17:32:29.947811 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.947689 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-credential-socket\") pod \"488ffbe5-4dff-4462-bdde-4b207a3fa156\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " Sep 18 17:32:29.947811 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.947732 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-token\") pod \"488ffbe5-4dff-4462-bdde-4b207a3fa156\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " Sep 18 17:32:29.947811 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.947759 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488ffbe5-4dff-4462-bdde-4b207a3fa156-istiod-ca-cert\") pod \"488ffbe5-4dff-4462-bdde-4b207a3fa156\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " Sep 18 17:32:29.947811 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.947800 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv2pp\" (UniqueName: \"kubernetes.io/projected/488ffbe5-4dff-4462-bdde-4b207a3fa156-kube-api-access-sv2pp\") pod \"488ffbe5-4dff-4462-bdde-4b207a3fa156\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " Sep 18 17:32:29.948130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.947864 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-workload-socket\") pod \"488ffbe5-4dff-4462-bdde-4b207a3fa156\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " Sep 18 17:32:29.948130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.947892 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-data" (OuterVolumeSpecName: "istio-data") pod "488ffbe5-4dff-4462-bdde-4b207a3fa156" (UID: "488ffbe5-4dff-4462-bdde-4b207a3fa156"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:32:29.948130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.948004 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-envoy\") pod \"488ffbe5-4dff-4462-bdde-4b207a3fa156\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " Sep 18 17:32:29.948130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.948048 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "488ffbe5-4dff-4462-bdde-4b207a3fa156" (UID: "488ffbe5-4dff-4462-bdde-4b207a3fa156"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:32:29.948130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.948071 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-workload-certs\") pod \"488ffbe5-4dff-4462-bdde-4b207a3fa156\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " Sep 18 17:32:29.948130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.948103 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-podinfo\") pod \"488ffbe5-4dff-4462-bdde-4b207a3fa156\" (UID: \"488ffbe5-4dff-4462-bdde-4b207a3fa156\") " Sep 18 17:32:29.948806 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.948136 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "488ffbe5-4dff-4462-bdde-4b207a3fa156" (UID: "488ffbe5-4dff-4462-bdde-4b207a3fa156"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:32:29.948806 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.948539 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:29.948806 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.948560 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:29.948806 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.948573 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:29.949566 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.949022 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "488ffbe5-4dff-4462-bdde-4b207a3fa156" (UID: "488ffbe5-4dff-4462-bdde-4b207a3fa156"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:32:29.949566 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.949416 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488ffbe5-4dff-4462-bdde-4b207a3fa156-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "488ffbe5-4dff-4462-bdde-4b207a3fa156" (UID: "488ffbe5-4dff-4462-bdde-4b207a3fa156"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:32:29.950656 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.950630 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "488ffbe5-4dff-4462-bdde-4b207a3fa156" (UID: "488ffbe5-4dff-4462-bdde-4b207a3fa156"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:32:29.950859 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.950836 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-token" (OuterVolumeSpecName: "istio-token") pod "488ffbe5-4dff-4462-bdde-4b207a3fa156" (UID: "488ffbe5-4dff-4462-bdde-4b207a3fa156"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:32:29.951158 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.951140 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488ffbe5-4dff-4462-bdde-4b207a3fa156-kube-api-access-sv2pp" (OuterVolumeSpecName: "kube-api-access-sv2pp") pod "488ffbe5-4dff-4462-bdde-4b207a3fa156" (UID: "488ffbe5-4dff-4462-bdde-4b207a3fa156"). InnerVolumeSpecName "kube-api-access-sv2pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:32:29.951645 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.951623 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "488ffbe5-4dff-4462-bdde-4b207a3fa156" (UID: "488ffbe5-4dff-4462-bdde-4b207a3fa156"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:32:29.954282 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.954228 2560 scope.go:117] "RemoveContainer" containerID="3a46d40bec19cf492166764fcf526b137cd3eeed72461cc045408551ef5b89bd" Sep 18 17:32:29.964231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.964204 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:29.976960 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.976937 2560 scope.go:117] "RemoveContainer" containerID="711593217bcc3ae00aa17e5c77ad0f5480474896a32ac7bfc3434911fd801097" Sep 18 17:32:29.977342 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:32:29.977284 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711593217bcc3ae00aa17e5c77ad0f5480474896a32ac7bfc3434911fd801097\": container with ID starting with 711593217bcc3ae00aa17e5c77ad0f5480474896a32ac7bfc3434911fd801097 not found: ID does not exist" containerID="711593217bcc3ae00aa17e5c77ad0f5480474896a32ac7bfc3434911fd801097" Sep 18 17:32:29.977410 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.977335 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711593217bcc3ae00aa17e5c77ad0f5480474896a32ac7bfc3434911fd801097"} err="failed to get container status \"711593217bcc3ae00aa17e5c77ad0f5480474896a32ac7bfc3434911fd801097\": rpc error: code = NotFound desc = could not find container \"711593217bcc3ae00aa17e5c77ad0f5480474896a32ac7bfc3434911fd801097\": container with ID starting with 711593217bcc3ae00aa17e5c77ad0f5480474896a32ac7bfc3434911fd801097 not found: ID does not exist" Sep 18 17:32:29.977410 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.977354 2560 scope.go:117] "RemoveContainer" containerID="3a46d40bec19cf492166764fcf526b137cd3eeed72461cc045408551ef5b89bd" Sep 18 17:32:29.977604 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:32:29.977584 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a46d40bec19cf492166764fcf526b137cd3eeed72461cc045408551ef5b89bd\": container with ID starting with 3a46d40bec19cf492166764fcf526b137cd3eeed72461cc045408551ef5b89bd not found: ID does not exist" containerID="3a46d40bec19cf492166764fcf526b137cd3eeed72461cc045408551ef5b89bd" Sep 18 17:32:29.977649 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:29.977611 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a46d40bec19cf492166764fcf526b137cd3eeed72461cc045408551ef5b89bd"} err="failed to get container status \"3a46d40bec19cf492166764fcf526b137cd3eeed72461cc045408551ef5b89bd\": rpc error: code = NotFound desc = could not find container \"3a46d40bec19cf492166764fcf526b137cd3eeed72461cc045408551ef5b89bd\": container with ID starting with 3a46d40bec19cf492166764fcf526b137cd3eeed72461cc045408551ef5b89bd not found: ID does not exist" Sep 18 17:32:30.050209 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:30.050113 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:30.050209 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:30.050146 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488ffbe5-4dff-4462-bdde-4b207a3fa156-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:30.050209 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:30.050158 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv2pp\" (UniqueName: \"kubernetes.io/projected/488ffbe5-4dff-4462-bdde-4b207a3fa156-kube-api-access-sv2pp\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:30.050209 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:30.050170 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:30.050209 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:30.050182 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488ffbe5-4dff-4462-bdde-4b207a3fa156-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:30.050209 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:30.050193 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488ffbe5-4dff-4462-bdde-4b207a3fa156-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:30.062848 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:30.062816 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:30.268125 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:30.268086 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp"] Sep 18 17:32:30.273710 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:30.273679 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-97798a-openshift-default-68bc9d7dfb-cw5vp"] Sep 18 17:32:30.866019 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:30.865989 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" path="/var/lib/kubelet/pods/488ffbe5-4dff-4462-bdde-4b207a3fa156/volumes" Sep 18 17:32:31.492603 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.492541 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:32:31.565520 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.565485 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-workload-certs\") pod \"87251652-60d8-4833-a3ab-aab4d31e8063\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " Sep 18 17:32:31.565520 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.565528 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlzhn\" (UniqueName: \"kubernetes.io/projected/87251652-60d8-4833-a3ab-aab4d31e8063-kube-api-access-nlzhn\") pod \"87251652-60d8-4833-a3ab-aab4d31e8063\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " Sep 18 17:32:31.565800 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.565595 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-workload-socket\") pod \"87251652-60d8-4833-a3ab-aab4d31e8063\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " Sep 18 17:32:31.565800 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.565622 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-istio-data\") pod \"87251652-60d8-4833-a3ab-aab4d31e8063\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " Sep 18 17:32:31.565800 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.565651 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87251652-60d8-4833-a3ab-aab4d31e8063-istiod-ca-cert\") pod \"87251652-60d8-4833-a3ab-aab4d31e8063\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " Sep 18 17:32:31.565800 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.565701 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87251652-60d8-4833-a3ab-aab4d31e8063-istio-token\") pod \"87251652-60d8-4833-a3ab-aab4d31e8063\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " Sep 18 17:32:31.565800 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.565739 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-credential-socket\") pod \"87251652-60d8-4833-a3ab-aab4d31e8063\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " Sep 18 17:32:31.566069 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.565804 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87251652-60d8-4833-a3ab-aab4d31e8063-istio-podinfo\") pod \"87251652-60d8-4833-a3ab-aab4d31e8063\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " Sep 18 17:32:31.566069 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.565809 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "87251652-60d8-4833-a3ab-aab4d31e8063" (UID: "87251652-60d8-4833-a3ab-aab4d31e8063"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:32:31.566069 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.565844 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-istio-envoy\") pod \"87251652-60d8-4833-a3ab-aab4d31e8063\" (UID: \"87251652-60d8-4833-a3ab-aab4d31e8063\") " Sep 18 17:32:31.566069 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.565878 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "87251652-60d8-4833-a3ab-aab4d31e8063" (UID: "87251652-60d8-4833-a3ab-aab4d31e8063"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:32:31.566069 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.565910 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-istio-data" (OuterVolumeSpecName: "istio-data") pod "87251652-60d8-4833-a3ab-aab4d31e8063" (UID: "87251652-60d8-4833-a3ab-aab4d31e8063"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:32:31.566069 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.566032 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "87251652-60d8-4833-a3ab-aab4d31e8063" (UID: "87251652-60d8-4833-a3ab-aab4d31e8063"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:32:31.566387 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.566120 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87251652-60d8-4833-a3ab-aab4d31e8063-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "87251652-60d8-4833-a3ab-aab4d31e8063" (UID: "87251652-60d8-4833-a3ab-aab4d31e8063"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:32:31.566387 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.566231 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:31.566387 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.566249 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:31.566387 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.566262 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:31.566387 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.566274 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87251652-60d8-4833-a3ab-aab4d31e8063-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:31.566387 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.566286 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:31.567990 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.567959 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87251652-60d8-4833-a3ab-aab4d31e8063-istio-token" (OuterVolumeSpecName: "istio-token") pod "87251652-60d8-4833-a3ab-aab4d31e8063" (UID: "87251652-60d8-4833-a3ab-aab4d31e8063"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:32:31.568096 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.568042 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87251652-60d8-4833-a3ab-aab4d31e8063-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "87251652-60d8-4833-a3ab-aab4d31e8063" (UID: "87251652-60d8-4833-a3ab-aab4d31e8063"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:32:31.568278 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.568260 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87251652-60d8-4833-a3ab-aab4d31e8063-kube-api-access-nlzhn" (OuterVolumeSpecName: "kube-api-access-nlzhn") pod "87251652-60d8-4833-a3ab-aab4d31e8063" (UID: "87251652-60d8-4833-a3ab-aab4d31e8063"). InnerVolumeSpecName "kube-api-access-nlzhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:32:31.568278 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.568272 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "87251652-60d8-4833-a3ab-aab4d31e8063" (UID: "87251652-60d8-4833-a3ab-aab4d31e8063"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:32:31.667661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.667614 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzhn\" (UniqueName: \"kubernetes.io/projected/87251652-60d8-4833-a3ab-aab4d31e8063-kube-api-access-nlzhn\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:31.667661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.667653 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87251652-60d8-4833-a3ab-aab4d31e8063-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:31.667661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.667665 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87251652-60d8-4833-a3ab-aab4d31e8063-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:31.667906 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.667676 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87251652-60d8-4833-a3ab-aab4d31e8063-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:32:31.958591 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.958556 2560 generic.go:356] "Generic (PLEG): container finished" podID="87251652-60d8-4833-a3ab-aab4d31e8063" containerID="e8978fa03c3c2485b0f6f5808bfa17b3de644224ac912fdf480ecb55d269f644" exitCode=0 Sep 18 17:32:31.958999 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.958617 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" Sep 18 17:32:31.958999 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.958644 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" event={"ID":"87251652-60d8-4833-a3ab-aab4d31e8063","Type":"ContainerDied","Data":"e8978fa03c3c2485b0f6f5808bfa17b3de644224ac912fdf480ecb55d269f644"} Sep 18 17:32:31.958999 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.958696 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv" event={"ID":"87251652-60d8-4833-a3ab-aab4d31e8063","Type":"ContainerDied","Data":"4cd98eaf8e4f41822c0c4e23bd9a0b00f20971cc367abf5b122431c7c2c4e4d2"} Sep 18 17:32:31.958999 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.958718 2560 scope.go:117] "RemoveContainer" containerID="e8978fa03c3c2485b0f6f5808bfa17b3de644224ac912fdf480ecb55d269f644" Sep 18 17:32:31.968852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.968829 2560 scope.go:117] "RemoveContainer" containerID="c9b9be90f9abd24f3eac086660f45ab8bcfcf1f890dc0598c9dff6b3cad7d984" Sep 18 17:32:31.979221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.979202 2560 scope.go:117] "RemoveContainer" containerID="e8978fa03c3c2485b0f6f5808bfa17b3de644224ac912fdf480ecb55d269f644" Sep 18 17:32:31.979586 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:32:31.979561 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8978fa03c3c2485b0f6f5808bfa17b3de644224ac912fdf480ecb55d269f644\": container with ID starting with e8978fa03c3c2485b0f6f5808bfa17b3de644224ac912fdf480ecb55d269f644 not found: ID does not exist" containerID="e8978fa03c3c2485b0f6f5808bfa17b3de644224ac912fdf480ecb55d269f644" Sep 18 17:32:31.979671 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.979595 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8978fa03c3c2485b0f6f5808bfa17b3de644224ac912fdf480ecb55d269f644"} err="failed to get container status \"e8978fa03c3c2485b0f6f5808bfa17b3de644224ac912fdf480ecb55d269f644\": rpc error: code = NotFound desc = could not find container \"e8978fa03c3c2485b0f6f5808bfa17b3de644224ac912fdf480ecb55d269f644\": container with ID starting with e8978fa03c3c2485b0f6f5808bfa17b3de644224ac912fdf480ecb55d269f644 not found: ID does not exist" Sep 18 17:32:31.979671 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.979614 2560 scope.go:117] "RemoveContainer" containerID="c9b9be90f9abd24f3eac086660f45ab8bcfcf1f890dc0598c9dff6b3cad7d984" Sep 18 17:32:31.979876 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:32:31.979859 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b9be90f9abd24f3eac086660f45ab8bcfcf1f890dc0598c9dff6b3cad7d984\": container with ID starting with c9b9be90f9abd24f3eac086660f45ab8bcfcf1f890dc0598c9dff6b3cad7d984 not found: ID does not exist" containerID="c9b9be90f9abd24f3eac086660f45ab8bcfcf1f890dc0598c9dff6b3cad7d984" Sep 18 17:32:31.979916 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.979882 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b9be90f9abd24f3eac086660f45ab8bcfcf1f890dc0598c9dff6b3cad7d984"} err="failed to get container status \"c9b9be90f9abd24f3eac086660f45ab8bcfcf1f890dc0598c9dff6b3cad7d984\": rpc error: code = NotFound desc = could not find container \"c9b9be90f9abd24f3eac086660f45ab8bcfcf1f890dc0598c9dff6b3cad7d984\": container with ID starting with c9b9be90f9abd24f3eac086660f45ab8bcfcf1f890dc0598c9dff6b3cad7d984 not found: ID does not exist" Sep 18 17:32:31.985110 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.985084 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv"] Sep 18 17:32:31.989646 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:31.989615 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-55a2c1b6-openshift-default-8649ffbf68-s46gv"] Sep 18 17:32:32.865492 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:32.865459 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" path="/var/lib/kubelet/pods/87251652-60d8-4833-a3ab-aab4d31e8063/volumes" Sep 18 17:32:33.119973 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:33.119893 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:33.224781 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:33.224746 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:33.325614 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:33.325577 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:33.480019 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:33.479988 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:38.315704 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:38.315665 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:32:39.842372 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:39.842344 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:39.954669 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:39.954638 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:40.840745 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:40.840713 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:41.218797 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.218763 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn"] Sep 18 17:32:41.219245 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.219233 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerName="istio-proxy" Sep 18 17:32:41.219303 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.219246 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerName="istio-proxy" Sep 18 17:32:41.219303 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.219261 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" containerName="istio-proxy" Sep 18 17:32:41.219303 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.219267 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" containerName="istio-proxy" Sep 18 17:32:41.219303 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.219276 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerName="istio-proxy" Sep 18 17:32:41.219303 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.219282 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerName="istio-proxy" Sep 18 17:32:41.219464 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.219289 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" containerName="istio-proxy" Sep 18 17:32:41.219464 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.219321 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" containerName="istio-proxy" Sep 18 17:32:41.219464 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.219391 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerName="istio-proxy" Sep 18 17:32:41.219464 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.219399 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" containerName="istio-proxy" Sep 18 17:32:41.219464 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.219407 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="488ffbe5-4dff-4462-bdde-4b207a3fa156" containerName="istio-proxy" Sep 18 17:32:41.219464 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.219416 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="87251652-60d8-4833-a3ab-aab4d31e8063" containerName="istio-proxy" Sep 18 17:32:41.221607 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.221591 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.228188 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.228071 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-dd235c-openshift-default-dockercfg-9dcv8\"" Sep 18 17:32:41.262880 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.262842 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn"] Sep 18 17:32:41.263159 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.263118 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:41.284321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.275610 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-credential-socket\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.284321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.275718 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-envoy\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.284321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.275774 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-token\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.284321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.275853 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istiod-ca-cert\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.284321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.275905 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5q59\" (UniqueName: \"kubernetes.io/projected/88400c22-6a60-4332-8a6e-c2d8d97eb76e-kube-api-access-q5q59\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.284321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.275937 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-data\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.284321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.275960 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-podinfo\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.284321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.275994 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-workload-socket\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.284321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.276024 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-workload-certs\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.377421 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.377387 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-credential-socket\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.377599 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.377454 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-envoy\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.377599 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.377492 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-token\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.377599 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.377546 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istiod-ca-cert\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.377599 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.377577 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5q59\" (UniqueName: \"kubernetes.io/projected/88400c22-6a60-4332-8a6e-c2d8d97eb76e-kube-api-access-q5q59\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.377832 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.377654 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-data\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.377832 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.377692 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-podinfo\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.377832 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.377729 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-workload-socket\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.377832 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.377763 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-workload-certs\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.378076 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.377848 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-credential-socket\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.378076 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.378000 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-data\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.378195 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.378140 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-workload-certs\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.378236 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.378193 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-workload-socket\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.378435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.378408 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istiod-ca-cert\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.379990 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.379927 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-envoy\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.380265 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.380243 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-podinfo\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.403074 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.403039 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5q59\" (UniqueName: \"kubernetes.io/projected/88400c22-6a60-4332-8a6e-c2d8d97eb76e-kube-api-access-q5q59\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.403226 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.403211 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-token\") pod \"e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.532048 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.531951 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:41.663689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.663645 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:41.824197 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.824172 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn"] Sep 18 17:32:41.825996 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:32:41.825965 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88400c22_6a60_4332_8a6e_c2d8d97eb76e.slice/crio-64ea62d5367369f56ca3cc206b806effa8fd2f76fe1fbf30168d4d18102658bf WatchSource:0}: Error finding container 64ea62d5367369f56ca3cc206b806effa8fd2f76fe1fbf30168d4d18102658bf: Status 404 returned error can't find the container with id 64ea62d5367369f56ca3cc206b806effa8fd2f76fe1fbf30168d4d18102658bf Sep 18 17:32:41.828431 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.828398 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:32:41.828488 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.828469 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:32:41.828522 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:41.828515 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:32:42.027662 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:42.027629 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" event={"ID":"88400c22-6a60-4332-8a6e-c2d8d97eb76e","Type":"ContainerStarted","Data":"0286f57963d9e07f55799cb22d4950a5a4231ebf17ffdb0e5dae3bc7cb266e00"} Sep 18 17:32:42.027845 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:42.027692 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" event={"ID":"88400c22-6a60-4332-8a6e-c2d8d97eb76e","Type":"ContainerStarted","Data":"64ea62d5367369f56ca3cc206b806effa8fd2f76fe1fbf30168d4d18102658bf"} Sep 18 17:32:42.079576 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:42.079458 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" podStartSLOduration=1.079434633 podStartE2EDuration="1.079434633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:32:42.077143238 +0000 UTC m=+1075.330304076" watchObservedRunningTime="2026-09-18 17:32:42.079434633 +0000 UTC m=+1075.332595451" Sep 18 17:32:42.533050 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:42.533010 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:42.537750 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:42.537725 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:43.033849 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:43.033816 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:43.034828 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:43.034810 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:32:43.637451 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:43.637415 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:44.282511 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:44.282478 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:44.446214 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:44.446181 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:44.846716 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:44.846680 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:46.175143 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:46.175108 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:47.204854 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:47.204824 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:48.550037 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:48.550001 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:49.584851 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:49.584788 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:49.686409 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:49.686375 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:49.819819 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:49.819775 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:50.216468 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:50.216438 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:32:50.374472 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:32:50.374438 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:01.528752 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:01.528716 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:03.303651 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:03.303599 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:04.828207 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:04.828167 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:33:12.685235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:12.685182 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:13.469572 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:13.469540 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:14.076522 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:14.076482 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 18 17:33:14.077023 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:14.076569 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 18 17:33:14.214969 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:14.214913 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:14.339413 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:14.339323 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:14.528937 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:14.528901 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:14.927209 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:14.927173 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc"] Sep 18 17:33:14.930942 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:14.930924 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:14.934720 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:14.934698 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-66efa6-openshift-default-dockercfg-xqdmf\"" Sep 18 17:33:14.941852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:14.941825 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:14.954724 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:14.954644 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc"] Sep 18 17:33:15.047409 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.045964 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-workload-socket\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.047409 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.046029 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-data\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.047409 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.046112 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-workload-certs\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.047409 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.046142 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-token\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.047409 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.046184 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-credential-socket\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.047409 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.046236 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd9ck\" (UniqueName: \"kubernetes.io/projected/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-kube-api-access-cd9ck\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.047409 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.046267 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-podinfo\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.047409 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.046360 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istiod-ca-cert\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.047409 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.046423 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-envoy\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.147802 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.147769 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-envoy\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.148314 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.147817 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-workload-socket\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.148314 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.147855 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-data\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.148314 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.147931 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-workload-certs\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.148314 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.147964 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-token\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.148314 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.148003 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-credential-socket\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.148314 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.148057 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd9ck\" (UniqueName: \"kubernetes.io/projected/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-kube-api-access-cd9ck\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.148314 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.148087 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-podinfo\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.148314 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.148161 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istiod-ca-cert\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.148776 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.148674 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-credential-socket\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.148989 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.148912 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-workload-socket\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.148989 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.148972 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istiod-ca-cert\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.149179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.149150 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-data\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.149451 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.149426 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-workload-certs\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.151824 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.151785 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-envoy\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.152074 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.152055 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-podinfo\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.159425 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.159402 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd9ck\" (UniqueName: \"kubernetes.io/projected/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-kube-api-access-cd9ck\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.178282 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.178223 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-token\") pod \"e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.241772 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.241730 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:15.395647 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.395615 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:15.495773 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.495695 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:15.532785 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.532759 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc"] Sep 18 17:33:15.534183 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:33:15.534155 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6408dd_e3ae_4a25_b34c_af0a8d3b0183.slice/crio-ee9009367e36f26916815ff1e2b6fb6d4bb891f890873a5c3c67aa6dd4563abb WatchSource:0}: Error finding container ee9009367e36f26916815ff1e2b6fb6d4bb891f890873a5c3c67aa6dd4563abb: Status 404 returned error can't find the container with id ee9009367e36f26916815ff1e2b6fb6d4bb891f890873a5c3c67aa6dd4563abb Sep 18 17:33:15.536852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.536811 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:15.536935 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.536906 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:15.536989 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:15.536958 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:16.143484 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:16.143441 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:16.234504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:16.234422 2560 generic.go:356] "Generic (PLEG): container finished" podID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerID="0286f57963d9e07f55799cb22d4950a5a4231ebf17ffdb0e5dae3bc7cb266e00" exitCode=0 Sep 18 17:33:16.234909 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:16.234504 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" event={"ID":"88400c22-6a60-4332-8a6e-c2d8d97eb76e","Type":"ContainerDied","Data":"0286f57963d9e07f55799cb22d4950a5a4231ebf17ffdb0e5dae3bc7cb266e00"} Sep 18 17:33:16.235069 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:16.235048 2560 scope.go:117] "RemoveContainer" containerID="0286f57963d9e07f55799cb22d4950a5a4231ebf17ffdb0e5dae3bc7cb266e00" Sep 18 17:33:16.236197 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:16.236173 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" event={"ID":"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183","Type":"ContainerStarted","Data":"52259bd42f462191dd29014f4df106e157a69da708316323d089558dac0459b8"} Sep 18 17:33:16.236328 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:16.236215 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" event={"ID":"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183","Type":"ContainerStarted","Data":"ee9009367e36f26916815ff1e2b6fb6d4bb891f890873a5c3c67aa6dd4563abb"} Sep 18 17:33:16.236393 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:16.236337 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:16.236431 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:16.236405 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:16.236466 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:16.236440 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:16.242794 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:16.242774 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:16.249439 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:16.249419 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:16.278405 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:16.277585 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" podStartSLOduration=2.277567736 podStartE2EDuration="2.277567736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:33:16.275680181 +0000 UTC m=+1109.528841031" watchObservedRunningTime="2026-09-18 17:33:16.277567736 +0000 UTC m=+1109.530728553" Sep 18 17:33:17.163396 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:17.163334 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:17.247648 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:17.246654 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" event={"ID":"88400c22-6a60-4332-8a6e-c2d8d97eb76e","Type":"ContainerStarted","Data":"99ee36409003a80576f0c5de1663ee42dd62e058d9a59f3baedcbfb290f23a4e"} Sep 18 17:33:17.247648 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:17.246741 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:17.253911 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:17.253871 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:17.532854 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:17.532758 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:33:17.533112 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:17.533093 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 18 17:33:17.533250 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:17.533233 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 18 17:33:17.933470 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:17.933437 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:18.183253 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:18.183216 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:18.274111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:18.274012 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:18.533337 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:18.533224 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 18 17:33:18.533337 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:18.533275 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 18 17:33:18.834584 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:18.834489 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:19.532902 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:19.532867 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 18 17:33:19.533348 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:19.532926 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 18 17:33:19.747789 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:19.747756 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:20.427839 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:20.427800 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:20.532843 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:20.532804 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 18 17:33:20.533033 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:20.532874 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 18 17:33:21.222793 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:21.222761 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:21.533044 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:21.532951 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 18 17:33:21.533044 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:21.533014 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 18 17:33:21.934803 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:21.934753 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:22.035823 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:22.035786 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:22.406443 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:22.406398 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:22.508262 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:22.508186 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:22.532462 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:22.532432 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 18 17:33:22.532635 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:22.532484 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 18 17:33:22.689696 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:22.689564 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:22.824716 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:22.824682 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:23.536254 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:23.536222 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:33:23.536552 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:23.536531 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:33:23.537162 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:23.537143 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:33:23.753049 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:23.753016 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:24.451861 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:24.451832 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:26.641118 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:26.641037 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:27.172078 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.172036 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2"] Sep 18 17:33:27.172377 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.172345 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" podUID="33fedc34-3591-43fb-a673-66fcdfe3a18c" containerName="maas-api" containerID="cri-o://cf2a6799c86c1e0caadda32b9ed2f681a81eb8a46f48dab89862f8de949c4522" gracePeriod=30 Sep 18 17:33:27.200873 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.200825 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk"] Sep 18 17:33:27.203530 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.203506 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:27.204743 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.204724 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-636b80-29829210-9kgvk"] Sep 18 17:33:27.277495 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.277454 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:27.324220 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.324188 2560 generic.go:356] "Generic (PLEG): container finished" podID="33fedc34-3591-43fb-a673-66fcdfe3a18c" containerID="cf2a6799c86c1e0caadda32b9ed2f681a81eb8a46f48dab89862f8de949c4522" exitCode=0 Sep 18 17:33:27.324397 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.324257 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" event={"ID":"33fedc34-3591-43fb-a673-66fcdfe3a18c","Type":"ContainerDied","Data":"cf2a6799c86c1e0caadda32b9ed2f681a81eb8a46f48dab89862f8de949c4522"} Sep 18 17:33:27.342567 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.342540 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:27.426219 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.426158 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:27.435413 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.435394 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:33:27.594542 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.594506 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-metrics-certs\") pod \"33fedc34-3591-43fb-a673-66fcdfe3a18c\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " Sep 18 17:33:27.594689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.594595 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5zsr\" (UniqueName: \"kubernetes.io/projected/33fedc34-3591-43fb-a673-66fcdfe3a18c-kube-api-access-w5zsr\") pod \"33fedc34-3591-43fb-a673-66fcdfe3a18c\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " Sep 18 17:33:27.594689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.594637 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-maas-api-tls\") pod \"33fedc34-3591-43fb-a673-66fcdfe3a18c\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " Sep 18 17:33:27.594689 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.594665 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fedc34-3591-43fb-a673-66fcdfe3a18c-openshift-service-ca\") pod \"33fedc34-3591-43fb-a673-66fcdfe3a18c\" (UID: \"33fedc34-3591-43fb-a673-66fcdfe3a18c\") " Sep 18 17:33:27.595115 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.595085 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33fedc34-3591-43fb-a673-66fcdfe3a18c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33fedc34-3591-43fb-a673-66fcdfe3a18c" (UID: "33fedc34-3591-43fb-a673-66fcdfe3a18c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:33:27.596787 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.596762 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "33fedc34-3591-43fb-a673-66fcdfe3a18c" (UID: "33fedc34-3591-43fb-a673-66fcdfe3a18c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:33:27.596869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.596796 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33fedc34-3591-43fb-a673-66fcdfe3a18c-kube-api-access-w5zsr" (OuterVolumeSpecName: "kube-api-access-w5zsr") pod "33fedc34-3591-43fb-a673-66fcdfe3a18c" (UID: "33fedc34-3591-43fb-a673-66fcdfe3a18c"). InnerVolumeSpecName "kube-api-access-w5zsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:33:27.596869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.596834 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33fedc34-3591-43fb-a673-66fcdfe3a18c" (UID: "33fedc34-3591-43fb-a673-66fcdfe3a18c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:33:27.696650 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.696573 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-metrics-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:27.696650 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.696600 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5zsr\" (UniqueName: \"kubernetes.io/projected/33fedc34-3591-43fb-a673-66fcdfe3a18c-kube-api-access-w5zsr\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:27.696650 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.696611 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33fedc34-3591-43fb-a673-66fcdfe3a18c-maas-api-tls\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:27.696650 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.696619 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fedc34-3591-43fb-a673-66fcdfe3a18c-openshift-service-ca\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:27.732450 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.732421 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:27.906556 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:27.906522 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:28.332319 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:28.332270 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" Sep 18 17:33:28.332319 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:28.332310 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2" event={"ID":"33fedc34-3591-43fb-a673-66fcdfe3a18c","Type":"ContainerDied","Data":"a10b22067888815d5da6e8b51ccb90a688b78e97f81a73e0e5c2b6b962b062af"} Sep 18 17:33:28.332530 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:28.332373 2560 scope.go:117] "RemoveContainer" containerID="cf2a6799c86c1e0caadda32b9ed2f681a81eb8a46f48dab89862f8de949c4522" Sep 18 17:33:28.357014 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:28.356975 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2"] Sep 18 17:33:28.359490 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:28.359465 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-636b80-75fbf6b758-8j2z2"] Sep 18 17:33:28.579655 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:28.579619 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:28.865951 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:28.865918 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33fedc34-3591-43fb-a673-66fcdfe3a18c" path="/var/lib/kubelet/pods/33fedc34-3591-43fb-a673-66fcdfe3a18c/volumes" Sep 18 17:33:28.866339 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:28.866231 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9cf2806-1599-4e12-9c0f-4e82a5a029ef" path="/var/lib/kubelet/pods/f9cf2806-1599-4e12-9c0f-4e82a5a029ef/volumes" Sep 18 17:33:29.264440 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:29.264411 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:30.035505 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:30.035473 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:30.743216 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:30.743187 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:31.639718 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:31.639677 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:32.814739 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:32.814703 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:32.929860 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:32.929824 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:33:34.609282 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:34.609248 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:34.710338 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:34.710278 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:34.885435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:34.885347 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj"] Sep 18 17:33:34.885924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:34.885897 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerName="istio-proxy" containerID="cri-o://f54fbb96852e2587307c6c8b661324856744da47848aea371ad2e6be42b24574" gracePeriod=30 Sep 18 17:33:34.899544 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:34.899518 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:35.213324 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:35.213272 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw"] Sep 18 17:33:35.213848 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:35.213833 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33fedc34-3591-43fb-a673-66fcdfe3a18c" containerName="maas-api" Sep 18 17:33:35.213899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:35.213851 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="33fedc34-3591-43fb-a673-66fcdfe3a18c" containerName="maas-api" Sep 18 17:33:35.213934 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:35.213924 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="33fedc34-3591-43fb-a673-66fcdfe3a18c" containerName="maas-api" Sep 18 17:33:35.216952 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:35.216934 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw" Sep 18 17:33:35.219672 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:35.219653 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\"" Sep 18 17:33:35.224546 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:35.224523 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw"] Sep 18 17:33:35.387549 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:35.387521 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/801eec33-acfe-44c7-ad09-5e198b302537-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw\" (UID: \"801eec33-acfe-44c7-ad09-5e198b302537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw" Sep 18 17:33:35.398821 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:35.398795 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:35.488569 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:35.488484 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/801eec33-acfe-44c7-ad09-5e198b302537-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw\" (UID: \"801eec33-acfe-44c7-ad09-5e198b302537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw" Sep 18 17:33:35.489123 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:35.489099 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/801eec33-acfe-44c7-ad09-5e198b302537-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw\" (UID: \"801eec33-acfe-44c7-ad09-5e198b302537\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw" Sep 18 17:33:35.531670 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:35.531632 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw" Sep 18 17:33:35.833485 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:35.833461 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw"] Sep 18 17:33:35.835313 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:33:35.835263 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801eec33_acfe_44c7_ad09_5e198b302537.slice/crio-4a0e165379e5d15d31d401f94b499f350097e5ca8cee94cc25dff718965b97b3 WatchSource:0}: Error finding container 4a0e165379e5d15d31d401f94b499f350097e5ca8cee94cc25dff718965b97b3: Status 404 returned error can't find the container with id 4a0e165379e5d15d31d401f94b499f350097e5ca8cee94cc25dff718965b97b3 Sep 18 17:33:36.351335 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:36.351223 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:36.391930 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:36.391891 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw" event={"ID":"801eec33-acfe-44c7-ad09-5e198b302537","Type":"ContainerStarted","Data":"d64082a3e897f9b41a0dae8ddaf1c1a1ffee6af1eccacabe1bbad1d2336ee88d"} Sep 18 17:33:36.392106 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:36.391964 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw" event={"ID":"801eec33-acfe-44c7-ad09-5e198b302537","Type":"ContainerStarted","Data":"4a0e165379e5d15d31d401f94b499f350097e5ca8cee94cc25dff718965b97b3"} Sep 18 17:33:36.410375 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:36.410311 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw" podStartSLOduration=1.41027431 podStartE2EDuration="1.41027431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:33:36.407858795 +0000 UTC m=+1129.661019637" watchObservedRunningTime="2026-09-18 17:33:36.41027431 +0000 UTC m=+1129.663435128" Sep 18 17:33:37.115823 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:37.115793 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:37.399228 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:37.399193 2560 generic.go:356] "Generic (PLEG): container finished" podID="801eec33-acfe-44c7-ad09-5e198b302537" containerID="d64082a3e897f9b41a0dae8ddaf1c1a1ffee6af1eccacabe1bbad1d2336ee88d" exitCode=0 Sep 18 17:33:37.399422 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:37.399272 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw" event={"ID":"801eec33-acfe-44c7-ad09-5e198b302537","Type":"ContainerDied","Data":"d64082a3e897f9b41a0dae8ddaf1c1a1ffee6af1eccacabe1bbad1d2336ee88d"} Sep 18 17:33:37.827627 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:37.827540 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:38.312496 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:38.312466 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:38.600119 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:38.600099 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw" Sep 18 17:33:38.625722 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:38.625686 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/801eec33-acfe-44c7-ad09-5e198b302537-maas-api-service-ca\") pod \"801eec33-acfe-44c7-ad09-5e198b302537\" (UID: \"801eec33-acfe-44c7-ad09-5e198b302537\") " Sep 18 17:33:38.626637 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:38.626611 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801eec33-acfe-44c7-ad09-5e198b302537-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "801eec33-acfe-44c7-ad09-5e198b302537" (UID: "801eec33-acfe-44c7-ad09-5e198b302537"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:33:38.727531 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:38.727492 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/801eec33-acfe-44c7-ad09-5e198b302537-maas-api-service-ca\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:38.797497 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:38.797465 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:39.414546 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:39.414505 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw" Sep 18 17:33:39.414930 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:39.414522 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw" event={"ID":"801eec33-acfe-44c7-ad09-5e198b302537","Type":"ContainerDied","Data":"4a0e165379e5d15d31d401f94b499f350097e5ca8cee94cc25dff718965b97b3"} Sep 18 17:33:39.414930 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:39.414642 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a0e165379e5d15d31d401f94b499f350097e5ca8cee94cc25dff718965b97b3" Sep 18 17:33:39.524735 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:39.524703 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:40.161471 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.161444 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:33:40.242919 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.242873 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istiod-ca-cert\") pod \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " Sep 18 17:33:40.243111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.242942 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-workload-socket\") pod \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " Sep 18 17:33:40.243111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.242971 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-podinfo\") pod \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " Sep 18 17:33:40.243879 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.243020 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-data\") pod \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " Sep 18 17:33:40.243879 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.243594 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-envoy\") pod \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " Sep 18 17:33:40.243879 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.243622 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "088b4a2f-0f19-4224-a9b8-4108e29dcd8e" (UID: "088b4a2f-0f19-4224-a9b8-4108e29dcd8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:33:40.243879 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.243675 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-credential-socket\") pod \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " Sep 18 17:33:40.243879 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.243823 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nslr2\" (UniqueName: \"kubernetes.io/projected/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-kube-api-access-nslr2\") pod \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " Sep 18 17:33:40.243879 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.243859 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-token\") pod \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " Sep 18 17:33:40.245777 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.244287 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-workload-certs\") pod \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\" (UID: \"088b4a2f-0f19-4224-a9b8-4108e29dcd8e\") " Sep 18 17:33:40.245777 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.245091 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:40.245777 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.245325 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "088b4a2f-0f19-4224-a9b8-4108e29dcd8e" (UID: "088b4a2f-0f19-4224-a9b8-4108e29dcd8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:33:40.245777 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.245511 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "088b4a2f-0f19-4224-a9b8-4108e29dcd8e" (UID: "088b4a2f-0f19-4224-a9b8-4108e29dcd8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:33:40.246054 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.246025 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "088b4a2f-0f19-4224-a9b8-4108e29dcd8e" (UID: "088b4a2f-0f19-4224-a9b8-4108e29dcd8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:33:40.246582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.246554 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-data" (OuterVolumeSpecName: "istio-data") pod "088b4a2f-0f19-4224-a9b8-4108e29dcd8e" (UID: "088b4a2f-0f19-4224-a9b8-4108e29dcd8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:33:40.254803 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.254718 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "088b4a2f-0f19-4224-a9b8-4108e29dcd8e" (UID: "088b4a2f-0f19-4224-a9b8-4108e29dcd8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:33:40.258685 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.258659 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "088b4a2f-0f19-4224-a9b8-4108e29dcd8e" (UID: "088b4a2f-0f19-4224-a9b8-4108e29dcd8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:33:40.258685 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.258676 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-token" (OuterVolumeSpecName: "istio-token") pod "088b4a2f-0f19-4224-a9b8-4108e29dcd8e" (UID: "088b4a2f-0f19-4224-a9b8-4108e29dcd8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:33:40.258863 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.258661 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-kube-api-access-nslr2" (OuterVolumeSpecName: "kube-api-access-nslr2") pod "088b4a2f-0f19-4224-a9b8-4108e29dcd8e" (UID: "088b4a2f-0f19-4224-a9b8-4108e29dcd8e"). InnerVolumeSpecName "kube-api-access-nslr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:33:40.347331 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.346356 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:40.347331 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.346386 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:40.347331 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.346396 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:40.347331 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.346405 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nslr2\" (UniqueName: \"kubernetes.io/projected/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-kube-api-access-nslr2\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:40.347331 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.346415 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:40.347331 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.346423 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:40.347331 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.346432 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:40.347331 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.346439 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/088b4a2f-0f19-4224-a9b8-4108e29dcd8e-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:40.423905 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.423861 2560 generic.go:356] "Generic (PLEG): container finished" podID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerID="f54fbb96852e2587307c6c8b661324856744da47848aea371ad2e6be42b24574" exitCode=0 Sep 18 17:33:40.424283 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.423926 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" Sep 18 17:33:40.424283 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.423936 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" event={"ID":"088b4a2f-0f19-4224-a9b8-4108e29dcd8e","Type":"ContainerDied","Data":"f54fbb96852e2587307c6c8b661324856744da47848aea371ad2e6be42b24574"} Sep 18 17:33:40.424283 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.424033 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj" event={"ID":"088b4a2f-0f19-4224-a9b8-4108e29dcd8e","Type":"ContainerDied","Data":"1613a0d10b6ea056e6fd43362a43d57ce826951da176c25e9b867576b2bd7077"} Sep 18 17:33:40.424283 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.424062 2560 scope.go:117] "RemoveContainer" containerID="f54fbb96852e2587307c6c8b661324856744da47848aea371ad2e6be42b24574" Sep 18 17:33:40.429692 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.429667 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:40.435629 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.435608 2560 scope.go:117] "RemoveContainer" containerID="961654a2e9944787031aa0890ce1871040030c17c0e272dfe0faf25bed5349f3" Sep 18 17:33:40.446724 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.445548 2560 scope.go:117] "RemoveContainer" containerID="f54fbb96852e2587307c6c8b661324856744da47848aea371ad2e6be42b24574" Sep 18 17:33:40.446724 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:40.445874 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f54fbb96852e2587307c6c8b661324856744da47848aea371ad2e6be42b24574\": container with ID starting with f54fbb96852e2587307c6c8b661324856744da47848aea371ad2e6be42b24574 not found: ID does not exist" containerID="f54fbb96852e2587307c6c8b661324856744da47848aea371ad2e6be42b24574" Sep 18 17:33:40.446724 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.445906 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f54fbb96852e2587307c6c8b661324856744da47848aea371ad2e6be42b24574"} err="failed to get container status \"f54fbb96852e2587307c6c8b661324856744da47848aea371ad2e6be42b24574\": rpc error: code = NotFound desc = could not find container \"f54fbb96852e2587307c6c8b661324856744da47848aea371ad2e6be42b24574\": container with ID starting with f54fbb96852e2587307c6c8b661324856744da47848aea371ad2e6be42b24574 not found: ID does not exist" Sep 18 17:33:40.446724 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.445931 2560 scope.go:117] "RemoveContainer" containerID="961654a2e9944787031aa0890ce1871040030c17c0e272dfe0faf25bed5349f3" Sep 18 17:33:40.447416 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:40.447314 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961654a2e9944787031aa0890ce1871040030c17c0e272dfe0faf25bed5349f3\": container with ID starting with 961654a2e9944787031aa0890ce1871040030c17c0e272dfe0faf25bed5349f3 not found: ID does not exist" containerID="961654a2e9944787031aa0890ce1871040030c17c0e272dfe0faf25bed5349f3" Sep 18 17:33:40.447416 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.447344 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961654a2e9944787031aa0890ce1871040030c17c0e272dfe0faf25bed5349f3"} err="failed to get container status \"961654a2e9944787031aa0890ce1871040030c17c0e272dfe0faf25bed5349f3\": rpc error: code = NotFound desc = could not find container \"961654a2e9944787031aa0890ce1871040030c17c0e272dfe0faf25bed5349f3\": container with ID starting with 961654a2e9944787031aa0890ce1871040030c17c0e272dfe0faf25bed5349f3 not found: ID does not exist" Sep 18 17:33:40.449807 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.449766 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj"] Sep 18 17:33:40.454266 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.454246 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-636b80-openshift-default-b795874cb-s8bfj"] Sep 18 17:33:40.865912 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:40.865879 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" path="/var/lib/kubelet/pods/088b4a2f-0f19-4224-a9b8-4108e29dcd8e/volumes" Sep 18 17:33:41.126942 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:41.126864 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:41.923679 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:41.923638 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:42.933432 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:42.933390 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:43.088956 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.088862 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:43.446770 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.446737 2560 generic.go:356] "Generic (PLEG): container finished" podID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerID="52259bd42f462191dd29014f4df106e157a69da708316323d089558dac0459b8" exitCode=0 Sep 18 17:33:43.446975 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.446796 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" event={"ID":"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183","Type":"ContainerDied","Data":"52259bd42f462191dd29014f4df106e157a69da708316323d089558dac0459b8"} Sep 18 17:33:43.447353 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.447336 2560 scope.go:117] "RemoveContainer" containerID="52259bd42f462191dd29014f4df106e157a69da708316323d089558dac0459b8" Sep 18 17:33:43.448787 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.448748 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:43.448879 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.448839 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:43.448919 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.448878 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:43.646601 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.646571 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:43.822595 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.822506 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7723-cc65ab"] Sep 18 17:33:43.822990 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.822978 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerName="istio-proxy" Sep 18 17:33:43.823054 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.822991 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerName="istio-proxy" Sep 18 17:33:43.823054 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.823009 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerName="istio-proxy" Sep 18 17:33:43.823054 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.823014 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerName="istio-proxy" Sep 18 17:33:43.823054 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.823022 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801eec33-acfe-44c7-ad09-5e198b302537" containerName="revoke-keys" Sep 18 17:33:43.823054 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.823027 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="801eec33-acfe-44c7-ad09-5e198b302537" containerName="revoke-keys" Sep 18 17:33:43.823340 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.823096 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerName="istio-proxy" Sep 18 17:33:43.823340 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.823103 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="088b4a2f-0f19-4224-a9b8-4108e29dcd8e" containerName="istio-proxy" Sep 18 17:33:43.823340 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.823111 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="801eec33-acfe-44c7-ad09-5e198b302537" containerName="revoke-keys" Sep 18 17:33:43.825254 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.825239 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-cc65ab" Sep 18 17:33:43.827926 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.827906 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mjflw\"" Sep 18 17:33:43.835684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.835665 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7723-cc65ab"] Sep 18 17:33:43.884379 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.884349 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlpfb\" (UniqueName: \"kubernetes.io/projected/3af611db-1163-4514-9779-7df357c55db6-kube-api-access-vlpfb\") pod \"test-curl-7723-cc65ab\" (UID: \"3af611db-1163-4514-9779-7df357c55db6\") " pod="opendatahub/test-curl-7723-cc65ab" Sep 18 17:33:43.985843 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.985807 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpfb\" (UniqueName: \"kubernetes.io/projected/3af611db-1163-4514-9779-7df357c55db6-kube-api-access-vlpfb\") pod \"test-curl-7723-cc65ab\" (UID: \"3af611db-1163-4514-9779-7df357c55db6\") " pod="opendatahub/test-curl-7723-cc65ab" Sep 18 17:33:43.997791 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:43.997743 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpfb\" (UniqueName: \"kubernetes.io/projected/3af611db-1163-4514-9779-7df357c55db6-kube-api-access-vlpfb\") pod \"test-curl-7723-cc65ab\" (UID: \"3af611db-1163-4514-9779-7df357c55db6\") " pod="opendatahub/test-curl-7723-cc65ab" Sep 18 17:33:44.147515 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:44.147487 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-cc65ab" Sep 18 17:33:44.431115 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:44.431086 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7723-cc65ab"] Sep 18 17:33:44.432244 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:33:44.432222 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3af611db_1163_4514_9779_7df357c55db6.slice/crio-29fea24fd77151c13424a1979ddb06423b31fde61d8b41a34622f43c8154b979 WatchSource:0}: Error finding container 29fea24fd77151c13424a1979ddb06423b31fde61d8b41a34622f43c8154b979: Status 404 returned error can't find the container with id 29fea24fd77151c13424a1979ddb06423b31fde61d8b41a34622f43c8154b979 Sep 18 17:33:44.434138 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:44.434121 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 17:33:44.457679 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:44.457637 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7723-cc65ab" event={"ID":"3af611db-1163-4514-9779-7df357c55db6","Type":"ContainerStarted","Data":"29fea24fd77151c13424a1979ddb06423b31fde61d8b41a34622f43c8154b979"} Sep 18 17:33:44.459786 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:44.459763 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" event={"ID":"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183","Type":"ContainerStarted","Data":"0b1928dafa2c04115531bf8dff8fbdf0d8772de313fc10b7df22a59abef6761a"} Sep 18 17:33:44.501149 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:44.501123 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:45.242690 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:45.242661 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:45.243706 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:45.243672 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 18 17:33:45.243860 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:45.243745 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" podUID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 18 17:33:45.362884 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:45.362840 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:46.003196 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.003161 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq"] Sep 18 17:33:46.003672 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.003639 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" podUID="033aacb7-d7be-4cb3-b990-235483f5ddeb" containerName="maas-api" containerID="cri-o://8345add3af5587ce05a4febfdacd486194f66872436b8409a6b16fca14e7a981" gracePeriod=30 Sep 18 17:33:46.034863 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.034807 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:46.102520 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.102277 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:46.201709 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.201672 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:46.218814 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.218751 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw"] Sep 18 17:33:46.222011 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.221991 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2aa65a-bmqkw"] Sep 18 17:33:46.242362 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.242258 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 18 17:33:46.242570 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.242354 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" podUID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 18 17:33:46.303245 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.303125 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:46.330445 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.330409 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:33:46.419396 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.419364 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-maas-api-tls\") pod \"033aacb7-d7be-4cb3-b990-235483f5ddeb\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " Sep 18 17:33:46.419587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.419503 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6cw7\" (UniqueName: \"kubernetes.io/projected/033aacb7-d7be-4cb3-b990-235483f5ddeb-kube-api-access-x6cw7\") pod \"033aacb7-d7be-4cb3-b990-235483f5ddeb\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " Sep 18 17:33:46.419587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.419572 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-metrics-certs\") pod \"033aacb7-d7be-4cb3-b990-235483f5ddeb\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " Sep 18 17:33:46.419693 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.419614 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033aacb7-d7be-4cb3-b990-235483f5ddeb-openshift-service-ca\") pod \"033aacb7-d7be-4cb3-b990-235483f5ddeb\" (UID: \"033aacb7-d7be-4cb3-b990-235483f5ddeb\") " Sep 18 17:33:46.420176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.420126 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033aacb7-d7be-4cb3-b990-235483f5ddeb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "033aacb7-d7be-4cb3-b990-235483f5ddeb" (UID: "033aacb7-d7be-4cb3-b990-235483f5ddeb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:33:46.421973 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.421948 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "033aacb7-d7be-4cb3-b990-235483f5ddeb" (UID: "033aacb7-d7be-4cb3-b990-235483f5ddeb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:33:46.422160 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.422134 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "033aacb7-d7be-4cb3-b990-235483f5ddeb" (UID: "033aacb7-d7be-4cb3-b990-235483f5ddeb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:33:46.423019 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.422983 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033aacb7-d7be-4cb3-b990-235483f5ddeb-kube-api-access-x6cw7" (OuterVolumeSpecName: "kube-api-access-x6cw7") pod "033aacb7-d7be-4cb3-b990-235483f5ddeb" (UID: "033aacb7-d7be-4cb3-b990-235483f5ddeb"). InnerVolumeSpecName "kube-api-access-x6cw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:33:46.483273 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.483233 2560 generic.go:356] "Generic (PLEG): container finished" podID="033aacb7-d7be-4cb3-b990-235483f5ddeb" containerID="8345add3af5587ce05a4febfdacd486194f66872436b8409a6b16fca14e7a981" exitCode=0 Sep 18 17:33:46.483490 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.483279 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" Sep 18 17:33:46.483490 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.483331 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" event={"ID":"033aacb7-d7be-4cb3-b990-235483f5ddeb","Type":"ContainerDied","Data":"8345add3af5587ce05a4febfdacd486194f66872436b8409a6b16fca14e7a981"} Sep 18 17:33:46.483490 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.483394 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq" event={"ID":"033aacb7-d7be-4cb3-b990-235483f5ddeb","Type":"ContainerDied","Data":"690d503ce4da5ac152337c6e8af0a31ff672bf119cfb7b0f26f522eab3c6f4b0"} Sep 18 17:33:46.483490 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.483413 2560 scope.go:117] "RemoveContainer" containerID="8345add3af5587ce05a4febfdacd486194f66872436b8409a6b16fca14e7a981" Sep 18 17:33:46.495171 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.495151 2560 scope.go:117] "RemoveContainer" containerID="8345add3af5587ce05a4febfdacd486194f66872436b8409a6b16fca14e7a981" Sep 18 17:33:46.495535 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:46.495514 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8345add3af5587ce05a4febfdacd486194f66872436b8409a6b16fca14e7a981\": container with ID starting with 8345add3af5587ce05a4febfdacd486194f66872436b8409a6b16fca14e7a981 not found: ID does not exist" containerID="8345add3af5587ce05a4febfdacd486194f66872436b8409a6b16fca14e7a981" Sep 18 17:33:46.495632 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.495541 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8345add3af5587ce05a4febfdacd486194f66872436b8409a6b16fca14e7a981"} err="failed to get container status \"8345add3af5587ce05a4febfdacd486194f66872436b8409a6b16fca14e7a981\": rpc error: code = NotFound desc = could not find container \"8345add3af5587ce05a4febfdacd486194f66872436b8409a6b16fca14e7a981\": container with ID starting with 8345add3af5587ce05a4febfdacd486194f66872436b8409a6b16fca14e7a981 not found: ID does not exist" Sep 18 17:33:46.509079 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.509020 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq"] Sep 18 17:33:46.513114 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.513087 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2aa65a-57ff8b987d-hm8mq"] Sep 18 17:33:46.521734 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.521710 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6cw7\" (UniqueName: \"kubernetes.io/projected/033aacb7-d7be-4cb3-b990-235483f5ddeb-kube-api-access-x6cw7\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:46.521820 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.521736 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-metrics-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:46.521820 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.521752 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033aacb7-d7be-4cb3-b990-235483f5ddeb-openshift-service-ca\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:46.521820 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.521765 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033aacb7-d7be-4cb3-b990-235483f5ddeb-maas-api-tls\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:46.572388 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.571157 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:46.716784 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.716750 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:46.876133 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.876039 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="033aacb7-d7be-4cb3-b990-235483f5ddeb" path="/var/lib/kubelet/pods/033aacb7-d7be-4cb3-b990-235483f5ddeb/volumes" Sep 18 17:33:46.876638 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:46.876616 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="801eec33-acfe-44c7-ad09-5e198b302537" path="/var/lib/kubelet/pods/801eec33-acfe-44c7-ad09-5e198b302537/volumes" Sep 18 17:33:47.243177 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:47.243115 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 18 17:33:47.243430 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:47.243184 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" podUID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 18 17:33:47.261321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:47.256715 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:47.525116 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:47.524998 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:48.146869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:48.146717 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:48.242860 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:48.242825 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 18 17:33:48.243026 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:48.242902 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" podUID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 18 17:33:48.498815 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:48.498728 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7723-cc65ab" event={"ID":"3af611db-1163-4514-9779-7df357c55db6","Type":"ContainerStarted","Data":"60dcf7063118829dd1e78c26cb96299a2d163e59b2d666ecbc195c7b819118a3"} Sep 18 17:33:48.514586 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:48.514541 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7723-cc65ab" podStartSLOduration=2.095575706 podStartE2EDuration="5.514528034s" totalImagesPullingTime="3.418952328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:33:48.51264329 +0000 UTC m=+1141.765804108" watchObservedRunningTime="2026-09-18 17:33:48.514528034 +0000 UTC m=+1141.767688850" Sep 18 17:33:48.833251 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:48.833172 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:49.130000 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.129921 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7723-cc65ab"] Sep 18 17:33:49.134222 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.134199 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7723-cc65ab"] Sep 18 17:33:49.242656 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.242626 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 18 17:33:49.242817 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.242687 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" podUID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 18 17:33:49.252793 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.252763 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7723-a85401"] Sep 18 17:33:49.261315 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.253693 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="033aacb7-d7be-4cb3-b990-235483f5ddeb" containerName="maas-api" Sep 18 17:33:49.274428 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.274400 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="033aacb7-d7be-4cb3-b990-235483f5ddeb" containerName="maas-api" Sep 18 17:33:49.274674 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.274658 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af611db-1163-4514-9779-7df357c55db6" containerName="test-curl-7723-cc65ab" Sep 18 17:33:49.274796 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.274782 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af611db-1163-4514-9779-7df357c55db6" containerName="test-curl-7723-cc65ab" Sep 18 17:33:49.275040 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.275025 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="033aacb7-d7be-4cb3-b990-235483f5ddeb" containerName="maas-api" Sep 18 17:33:49.275151 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.275139 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af611db-1163-4514-9779-7df357c55db6" containerName="test-curl-7723-cc65ab" Sep 18 17:33:49.280440 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.280395 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7723-a85401"] Sep 18 17:33:49.280581 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.280494 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-a85401" Sep 18 17:33:49.375664 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.375624 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czd7r\" (UniqueName: \"kubernetes.io/projected/85643342-304c-4007-bbc7-0a14a139bb67-kube-api-access-czd7r\") pod \"test-curl-7723-a85401\" (UID: \"85643342-304c-4007-bbc7-0a14a139bb67\") " pod="opendatahub/test-curl-7723-a85401" Sep 18 17:33:49.476508 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.476474 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czd7r\" (UniqueName: \"kubernetes.io/projected/85643342-304c-4007-bbc7-0a14a139bb67-kube-api-access-czd7r\") pod \"test-curl-7723-a85401\" (UID: \"85643342-304c-4007-bbc7-0a14a139bb67\") " pod="opendatahub/test-curl-7723-a85401" Sep 18 17:33:49.485192 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.485162 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czd7r\" (UniqueName: \"kubernetes.io/projected/85643342-304c-4007-bbc7-0a14a139bb67-kube-api-access-czd7r\") pod \"test-curl-7723-a85401\" (UID: \"85643342-304c-4007-bbc7-0a14a139bb67\") " pod="opendatahub/test-curl-7723-a85401" Sep 18 17:33:49.504111 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:49.504088 2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7723-cc65ab" container="test-curl-7723-cc65ab" Sep 18 17:33:49.533269 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.533244 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:49.591524 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.591490 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-a85401" Sep 18 17:33:49.888165 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:49.888142 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7723-a85401"] Sep 18 17:33:49.890269 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:33:49.890241 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85643342_304c_4007_bbc7_0a14a139bb67.slice/crio-c1d9dd973380c33bf1c33fa76f60085f94fd0b2f1a352c715b03be6a590a5261 WatchSource:0}: Error finding container c1d9dd973380c33bf1c33fa76f60085f94fd0b2f1a352c715b03be6a590a5261: Status 404 returned error can't find the container with id c1d9dd973380c33bf1c33fa76f60085f94fd0b2f1a352c715b03be6a590a5261 Sep 18 17:33:50.246537 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:50.246507 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:50.247624 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:50.247605 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:33:50.510924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:50.510827 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7723-a85401" event={"ID":"85643342-304c-4007-bbc7-0a14a139bb67","Type":"ContainerStarted","Data":"4905c268feca69be548165fe8f73456017b4a6ec38c0cd363730e6a129c9e010"} Sep 18 17:33:50.510924 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:50.510879 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7723-a85401" event={"ID":"85643342-304c-4007-bbc7-0a14a139bb67","Type":"ContainerStarted","Data":"c1d9dd973380c33bf1c33fa76f60085f94fd0b2f1a352c715b03be6a590a5261"} Sep 18 17:33:50.511161 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:50.510919 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7723-cc65ab" podUID="3af611db-1163-4514-9779-7df357c55db6" containerName="test-curl-7723-cc65ab" containerID="cri-o://60dcf7063118829dd1e78c26cb96299a2d163e59b2d666ecbc195c7b819118a3" gracePeriod=2 Sep 18 17:33:50.526792 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:50.526735 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7723-a85401" podStartSLOduration=1.526719603 podStartE2EDuration="1.526719603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:33:50.524971068 +0000 UTC m=+1143.778131882" watchObservedRunningTime="2026-09-18 17:33:50.526719603 +0000 UTC m=+1143.779880420" Sep 18 17:33:50.526939 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:50.526897 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7723-cc65ab\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="3af611db-1163-4514-9779-7df357c55db6" pod="opendatahub/test-curl-7723-cc65ab" Sep 18 17:33:50.528805 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:50.528785 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7723-cc65ab\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="3af611db-1163-4514-9779-7df357c55db6" pod="opendatahub/test-curl-7723-cc65ab" Sep 18 17:33:51.102978 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.102942 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7723-a85401"] Sep 18 17:33:51.110404 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.110376 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7723-a85401"] Sep 18 17:33:51.218859 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.218830 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7723-510552"] Sep 18 17:33:51.219367 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.219353 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85643342-304c-4007-bbc7-0a14a139bb67" containerName="test-curl-7723-a85401" Sep 18 17:33:51.219418 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.219369 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="85643342-304c-4007-bbc7-0a14a139bb67" containerName="test-curl-7723-a85401" Sep 18 17:33:51.219454 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.219444 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="85643342-304c-4007-bbc7-0a14a139bb67" containerName="test-curl-7723-a85401" Sep 18 17:33:51.221492 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.221475 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-510552" Sep 18 17:33:51.229169 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.229134 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7723-510552"] Sep 18 17:33:51.298554 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.298525 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmsd2\" (UniqueName: \"kubernetes.io/projected/8d94ba19-fbc0-4523-b0da-8f42d7c19b24-kube-api-access-xmsd2\") pod \"test-curl-7723-510552\" (UID: \"8d94ba19-fbc0-4523-b0da-8f42d7c19b24\") " pod="opendatahub/test-curl-7723-510552" Sep 18 17:33:51.399808 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.399775 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmsd2\" (UniqueName: \"kubernetes.io/projected/8d94ba19-fbc0-4523-b0da-8f42d7c19b24-kube-api-access-xmsd2\") pod \"test-curl-7723-510552\" (UID: \"8d94ba19-fbc0-4523-b0da-8f42d7c19b24\") " pod="opendatahub/test-curl-7723-510552" Sep 18 17:33:51.409343 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.409314 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmsd2\" (UniqueName: \"kubernetes.io/projected/8d94ba19-fbc0-4523-b0da-8f42d7c19b24-kube-api-access-xmsd2\") pod \"test-curl-7723-510552\" (UID: \"8d94ba19-fbc0-4523-b0da-8f42d7c19b24\") " pod="opendatahub/test-curl-7723-510552" Sep 18 17:33:51.517384 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:51.517287 2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7723-a85401" container="test-curl-7723-a85401" Sep 18 17:33:51.534162 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.534144 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-510552" Sep 18 17:33:51.638185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.638155 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:51.803810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:51.803786 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7723-510552"] Sep 18 17:33:51.805498 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:33:51.805471 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d94ba19_fbc0_4523_b0da_8f42d7c19b24.slice/crio-7553b5d5c3e5786d3650073689cc194ebb5dbb63bc0e398879e66111f680047d WatchSource:0}: Error finding container 7553b5d5c3e5786d3650073689cc194ebb5dbb63bc0e398879e66111f680047d: Status 404 returned error can't find the container with id 7553b5d5c3e5786d3650073689cc194ebb5dbb63bc0e398879e66111f680047d Sep 18 17:33:52.523872 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:52.523839 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7723-510552" event={"ID":"8d94ba19-fbc0-4523-b0da-8f42d7c19b24","Type":"ContainerStarted","Data":"47cd29a393cf62007271e3b9a91e1c8e29bdece8402bb22012d96571850d65da"} Sep 18 17:33:52.524233 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:52.523886 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7723-510552" event={"ID":"8d94ba19-fbc0-4523-b0da-8f42d7c19b24","Type":"ContainerStarted","Data":"7553b5d5c3e5786d3650073689cc194ebb5dbb63bc0e398879e66111f680047d"} Sep 18 17:33:52.524233 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:52.523928 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7723-a85401" podUID="85643342-304c-4007-bbc7-0a14a139bb67" containerName="test-curl-7723-a85401" containerID="cri-o://4905c268feca69be548165fe8f73456017b4a6ec38c0cd363730e6a129c9e010" gracePeriod=2 Sep 18 17:33:52.526545 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:52.526519 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7723-a85401\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="85643342-304c-4007-bbc7-0a14a139bb67" pod="opendatahub/test-curl-7723-a85401" Sep 18 17:33:52.542824 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:52.542790 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7723-a85401\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="85643342-304c-4007-bbc7-0a14a139bb67" pod="opendatahub/test-curl-7723-a85401" Sep 18 17:33:52.543690 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:52.543648 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7723-510552" podStartSLOduration=1.543632487 podStartE2EDuration="1.543632487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:33:52.540831455 +0000 UTC m=+1145.793992273" watchObservedRunningTime="2026-09-18 17:33:52.543632487 +0000 UTC m=+1145.796793304" Sep 18 17:33:52.688403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:52.688379 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-cc65ab" Sep 18 17:33:52.691403 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:52.691377 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7723-a85401\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="85643342-304c-4007-bbc7-0a14a139bb67" pod="opendatahub/test-curl-7723-a85401" Sep 18 17:33:52.814657 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:52.814566 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlpfb\" (UniqueName: \"kubernetes.io/projected/3af611db-1163-4514-9779-7df357c55db6-kube-api-access-vlpfb\") pod \"3af611db-1163-4514-9779-7df357c55db6\" (UID: \"3af611db-1163-4514-9779-7df357c55db6\") " Sep 18 17:33:52.816736 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:52.816711 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af611db-1163-4514-9779-7df357c55db6-kube-api-access-vlpfb" (OuterVolumeSpecName: "kube-api-access-vlpfb") pod "3af611db-1163-4514-9779-7df357c55db6" (UID: "3af611db-1163-4514-9779-7df357c55db6"). InnerVolumeSpecName "kube-api-access-vlpfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:33:52.865345 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:52.865284 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af611db-1163-4514-9779-7df357c55db6" path="/var/lib/kubelet/pods/3af611db-1163-4514-9779-7df357c55db6/volumes" Sep 18 17:33:52.915872 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:52.915837 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlpfb\" (UniqueName: \"kubernetes.io/projected/3af611db-1163-4514-9779-7df357c55db6-kube-api-access-vlpfb\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:53.139200 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.136636 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7723-510552"] Sep 18 17:33:53.145612 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.145577 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7723-510552"] Sep 18 17:33:53.248468 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.248431 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7723-9c8020"] Sep 18 17:33:53.249191 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.249174 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d94ba19-fbc0-4523-b0da-8f42d7c19b24" containerName="test-curl-7723-510552" Sep 18 17:33:53.249241 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.249194 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d94ba19-fbc0-4523-b0da-8f42d7c19b24" containerName="test-curl-7723-510552" Sep 18 17:33:53.249316 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.249304 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d94ba19-fbc0-4523-b0da-8f42d7c19b24" containerName="test-curl-7723-510552" Sep 18 17:33:53.251511 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.251496 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-9c8020" Sep 18 17:33:53.259527 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.259506 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7723-9c8020"] Sep 18 17:33:53.420214 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.420183 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knf4z\" (UniqueName: \"kubernetes.io/projected/0e3c9ee3-a806-413f-9186-6c8695f0c5fa-kube-api-access-knf4z\") pod \"test-curl-7723-9c8020\" (UID: \"0e3c9ee3-a806-413f-9186-6c8695f0c5fa\") " pod="opendatahub/test-curl-7723-9c8020" Sep 18 17:33:53.521052 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.521012 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knf4z\" (UniqueName: \"kubernetes.io/projected/0e3c9ee3-a806-413f-9186-6c8695f0c5fa-kube-api-access-knf4z\") pod \"test-curl-7723-9c8020\" (UID: \"0e3c9ee3-a806-413f-9186-6c8695f0c5fa\") " pod="opendatahub/test-curl-7723-9c8020" Sep 18 17:33:53.529912 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.529884 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knf4z\" (UniqueName: \"kubernetes.io/projected/0e3c9ee3-a806-413f-9186-6c8695f0c5fa-kube-api-access-knf4z\") pod \"test-curl-7723-9c8020\" (UID: \"0e3c9ee3-a806-413f-9186-6c8695f0c5fa\") " pod="opendatahub/test-curl-7723-9c8020" Sep 18 17:33:53.531610 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.531582 2560 generic.go:356] "Generic (PLEG): container finished" podID="3af611db-1163-4514-9779-7df357c55db6" containerID="60dcf7063118829dd1e78c26cb96299a2d163e59b2d666ecbc195c7b819118a3" exitCode=137 Sep 18 17:33:53.531724 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.531635 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-cc65ab" Sep 18 17:33:53.531724 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.531679 2560 scope.go:117] "RemoveContainer" containerID="60dcf7063118829dd1e78c26cb96299a2d163e59b2d666ecbc195c7b819118a3" Sep 18 17:33:53.531847 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:53.531788 2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7723-510552" container="test-curl-7723-510552" Sep 18 17:33:53.546602 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.546418 2560 scope.go:117] "RemoveContainer" containerID="60dcf7063118829dd1e78c26cb96299a2d163e59b2d666ecbc195c7b819118a3" Sep 18 17:33:53.546794 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:53.546769 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60dcf7063118829dd1e78c26cb96299a2d163e59b2d666ecbc195c7b819118a3\": container with ID starting with 60dcf7063118829dd1e78c26cb96299a2d163e59b2d666ecbc195c7b819118a3 not found: ID does not exist" containerID="60dcf7063118829dd1e78c26cb96299a2d163e59b2d666ecbc195c7b819118a3" Sep 18 17:33:53.546856 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.546809 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60dcf7063118829dd1e78c26cb96299a2d163e59b2d666ecbc195c7b819118a3"} err="failed to get container status \"60dcf7063118829dd1e78c26cb96299a2d163e59b2d666ecbc195c7b819118a3\": rpc error: code = NotFound desc = could not find container \"60dcf7063118829dd1e78c26cb96299a2d163e59b2d666ecbc195c7b819118a3\": container with ID starting with 60dcf7063118829dd1e78c26cb96299a2d163e59b2d666ecbc195c7b819118a3 not found: ID does not exist" Sep 18 17:33:53.561965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.561940 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-9c8020" Sep 18 17:33:53.832121 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:53.832093 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7723-9c8020"] Sep 18 17:33:53.833631 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:33:53.833606 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e3c9ee3_a806_413f_9186_6c8695f0c5fa.slice/crio-6d3e74318e100b28587a4be6199a438b633aef7f1ce8485eb159a1a920b10215 WatchSource:0}: Error finding container 6d3e74318e100b28587a4be6199a438b633aef7f1ce8485eb159a1a920b10215: Status 404 returned error can't find the container with id 6d3e74318e100b28587a4be6199a438b633aef7f1ce8485eb159a1a920b10215 Sep 18 17:33:54.538620 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:54.538593 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7723-9c8020" event={"ID":"0e3c9ee3-a806-413f-9186-6c8695f0c5fa","Type":"ContainerStarted","Data":"688d4a77161936abd32d772c376574a5ed9b2306dab23081de4eed9cd3ca2c2f"} Sep 18 17:33:54.538962 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:54.538632 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7723-9c8020" event={"ID":"0e3c9ee3-a806-413f-9186-6c8695f0c5fa","Type":"ContainerStarted","Data":"6d3e74318e100b28587a4be6199a438b633aef7f1ce8485eb159a1a920b10215"} Sep 18 17:33:54.539820 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:54.539794 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7723-510552" podUID="8d94ba19-fbc0-4523-b0da-8f42d7c19b24" containerName="test-curl-7723-510552" containerID="cri-o://47cd29a393cf62007271e3b9a91e1c8e29bdece8402bb22012d96571850d65da" gracePeriod=2 Sep 18 17:33:54.555217 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:54.555168 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7723-9c8020" podStartSLOduration=1.555153399 podStartE2EDuration="1.555153399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:33:54.554629174 +0000 UTC m=+1147.807789993" watchObservedRunningTime="2026-09-18 17:33:54.555153399 +0000 UTC m=+1147.808314272" Sep 18 17:33:54.556632 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:54.556606 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7723-510552\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="8d94ba19-fbc0-4523-b0da-8f42d7c19b24" pod="opendatahub/test-curl-7723-510552" Sep 18 17:33:54.706489 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:54.706469 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-a85401" Sep 18 17:33:54.709163 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:54.709137 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7723-510552\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="8d94ba19-fbc0-4523-b0da-8f42d7c19b24" pod="opendatahub/test-curl-7723-510552" Sep 18 17:33:54.835127 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:54.835046 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czd7r\" (UniqueName: \"kubernetes.io/projected/85643342-304c-4007-bbc7-0a14a139bb67-kube-api-access-czd7r\") pod \"85643342-304c-4007-bbc7-0a14a139bb67\" (UID: \"85643342-304c-4007-bbc7-0a14a139bb67\") " Sep 18 17:33:54.837033 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:54.837011 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85643342-304c-4007-bbc7-0a14a139bb67-kube-api-access-czd7r" (OuterVolumeSpecName: "kube-api-access-czd7r") pod "85643342-304c-4007-bbc7-0a14a139bb67" (UID: "85643342-304c-4007-bbc7-0a14a139bb67"). InnerVolumeSpecName "kube-api-access-czd7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:33:54.865239 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:54.865209 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85643342-304c-4007-bbc7-0a14a139bb67" path="/var/lib/kubelet/pods/85643342-304c-4007-bbc7-0a14a139bb67/volumes" Sep 18 17:33:54.936376 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:54.936346 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czd7r\" (UniqueName: \"kubernetes.io/projected/85643342-304c-4007-bbc7-0a14a139bb67-kube-api-access-czd7r\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:55.132953 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:55.132866 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7723-9c8020"] Sep 18 17:33:55.138540 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:55.138510 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7723-9c8020"] Sep 18 17:33:55.546649 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:55.546611 2560 generic.go:356] "Generic (PLEG): container finished" podID="85643342-304c-4007-bbc7-0a14a139bb67" containerID="4905c268feca69be548165fe8f73456017b4a6ec38c0cd363730e6a129c9e010" exitCode=137 Sep 18 17:33:55.547103 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:55.546673 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-a85401" Sep 18 17:33:55.547103 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:55.546720 2560 scope.go:117] "RemoveContainer" containerID="4905c268feca69be548165fe8f73456017b4a6ec38c0cd363730e6a129c9e010" Sep 18 17:33:55.547103 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:55.546883 2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7723-9c8020" container="test-curl-7723-9c8020" Sep 18 17:33:55.555963 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:55.555945 2560 scope.go:117] "RemoveContainer" containerID="4905c268feca69be548165fe8f73456017b4a6ec38c0cd363730e6a129c9e010" Sep 18 17:33:55.556248 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:55.556229 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4905c268feca69be548165fe8f73456017b4a6ec38c0cd363730e6a129c9e010\": container with ID starting with 4905c268feca69be548165fe8f73456017b4a6ec38c0cd363730e6a129c9e010 not found: ID does not exist" containerID="4905c268feca69be548165fe8f73456017b4a6ec38c0cd363730e6a129c9e010" Sep 18 17:33:55.556382 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:55.556259 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4905c268feca69be548165fe8f73456017b4a6ec38c0cd363730e6a129c9e010"} err="failed to get container status \"4905c268feca69be548165fe8f73456017b4a6ec38c0cd363730e6a129c9e010\": rpc error: code = NotFound desc = could not find container \"4905c268feca69be548165fe8f73456017b4a6ec38c0cd363730e6a129c9e010\": container with ID starting with 4905c268feca69be548165fe8f73456017b4a6ec38c0cd363730e6a129c9e010 not found: ID does not exist" Sep 18 17:33:55.711676 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:55.711652 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:56.019229 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.019202 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k"] Sep 18 17:33:56.019707 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.019694 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3c9ee3-a806-413f-9186-6c8695f0c5fa" containerName="test-curl-7723-9c8020" Sep 18 17:33:56.019748 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.019708 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3c9ee3-a806-413f-9186-6c8695f0c5fa" containerName="test-curl-7723-9c8020" Sep 18 17:33:56.019794 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.019781 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e3c9ee3-a806-413f-9186-6c8695f0c5fa" containerName="test-curl-7723-9c8020" Sep 18 17:33:56.022945 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.022925 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.026605 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.026587 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b68d6946-openshift-default-dockercfg-97hcr\"" Sep 18 17:33:56.037029 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.037007 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k"] Sep 18 17:33:56.044157 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.044111 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:56.150983 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.150947 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppzkz\" (UniqueName: \"kubernetes.io/projected/58f4236b-5856-469a-bd03-76bfe6c2b3e1-kube-api-access-ppzkz\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.151156 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.151020 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-podinfo\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.151156 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.151061 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-credential-socket\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.151156 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.151096 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-envoy\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.151257 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.151219 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-data\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.151257 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.151247 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-workload-certs\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.151346 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.151280 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-token\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.151386 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.151355 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-workload-socket\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.151419 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.151405 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istiod-ca-cert\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.252931 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.252890 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-data\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.252931 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.252935 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-workload-certs\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.253154 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.252975 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-token\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.253154 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.253001 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-workload-socket\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.253154 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.253040 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istiod-ca-cert\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.253154 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.253099 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppzkz\" (UniqueName: \"kubernetes.io/projected/58f4236b-5856-469a-bd03-76bfe6c2b3e1-kube-api-access-ppzkz\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.253154 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.253146 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-podinfo\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.253435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.253179 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-credential-socket\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.253435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.253212 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-envoy\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.253681 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.253634 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-workload-socket\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.253880 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.253858 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-data\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.253961 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.253892 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-credential-socket\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.253961 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.253864 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istiod-ca-cert\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.253961 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.253946 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-workload-certs\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.256031 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.256008 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-envoy\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.256251 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.256232 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-podinfo\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.262323 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.262270 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-token\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.262700 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.262677 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppzkz\" (UniqueName: \"kubernetes.io/projected/58f4236b-5856-469a-bd03-76bfe6c2b3e1-kube-api-access-ppzkz\") pod \"e2e-mt-b68d6946-openshift-default-574bff878d-5j86k\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.329678 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.329606 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:56.332695 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.332668 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:56.554462 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.554418 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7723-9c8020" podUID="0e3c9ee3-a806-413f-9186-6c8695f0c5fa" containerName="test-curl-7723-9c8020" containerID="cri-o://688d4a77161936abd32d772c376574a5ed9b2306dab23081de4eed9cd3ca2c2f" gracePeriod=2 Sep 18 17:33:56.556836 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:56.556803 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7723-9c8020\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="0e3c9ee3-a806-413f-9186-6c8695f0c5fa" pod="opendatahub/test-curl-7723-9c8020" Sep 18 17:33:56.610733 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.610703 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k"] Sep 18 17:33:56.612247 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:33:56.612219 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f4236b_5856_469a_bd03_76bfe6c2b3e1.slice/crio-1393bf48205b1eb816f7ef228b26586712dff40fcf469426c74b8d91134f7223 WatchSource:0}: Error finding container 1393bf48205b1eb816f7ef228b26586712dff40fcf469426c74b8d91134f7223: Status 404 returned error can't find the container with id 1393bf48205b1eb816f7ef228b26586712dff40fcf469426c74b8d91134f7223 Sep 18 17:33:56.615013 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.614980 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:56.615113 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.615047 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:56.615113 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.615080 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:56.721698 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.721674 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-510552" Sep 18 17:33:56.723792 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:56.723764 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7723-9c8020\" is forbidden: User \"system:node:ip-10-0-128-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-9.ec2.internal' and this object" podUID="0e3c9ee3-a806-413f-9186-6c8695f0c5fa" pod="opendatahub/test-curl-7723-9c8020" Sep 18 17:33:56.859410 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.859324 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmsd2\" (UniqueName: \"kubernetes.io/projected/8d94ba19-fbc0-4523-b0da-8f42d7c19b24-kube-api-access-xmsd2\") pod \"8d94ba19-fbc0-4523-b0da-8f42d7c19b24\" (UID: \"8d94ba19-fbc0-4523-b0da-8f42d7c19b24\") " Sep 18 17:33:56.862274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.862241 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d94ba19-fbc0-4523-b0da-8f42d7c19b24-kube-api-access-xmsd2" (OuterVolumeSpecName: "kube-api-access-xmsd2") pod "8d94ba19-fbc0-4523-b0da-8f42d7c19b24" (UID: "8d94ba19-fbc0-4523-b0da-8f42d7c19b24"). InnerVolumeSpecName "kube-api-access-xmsd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:33:56.866047 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.866023 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d94ba19-fbc0-4523-b0da-8f42d7c19b24" path="/var/lib/kubelet/pods/8d94ba19-fbc0-4523-b0da-8f42d7c19b24/volumes" Sep 18 17:33:56.961199 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:56.961166 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmsd2\" (UniqueName: \"kubernetes.io/projected/8d94ba19-fbc0-4523-b0da-8f42d7c19b24-kube-api-access-xmsd2\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:57.562185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:57.562093 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" event={"ID":"58f4236b-5856-469a-bd03-76bfe6c2b3e1","Type":"ContainerStarted","Data":"f575c6113ed8a54210da49e946667546336027c4be06bf2626a880b13219ed5e"} Sep 18 17:33:57.562185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:57.562148 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" event={"ID":"58f4236b-5856-469a-bd03-76bfe6c2b3e1","Type":"ContainerStarted","Data":"1393bf48205b1eb816f7ef228b26586712dff40fcf469426c74b8d91134f7223"} Sep 18 17:33:57.563442 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:57.563414 2560 generic.go:356] "Generic (PLEG): container finished" podID="8d94ba19-fbc0-4523-b0da-8f42d7c19b24" containerID="47cd29a393cf62007271e3b9a91e1c8e29bdece8402bb22012d96571850d65da" exitCode=137 Sep 18 17:33:57.563569 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:57.563445 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-510552" Sep 18 17:33:57.563569 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:57.563481 2560 scope.go:117] "RemoveContainer" containerID="47cd29a393cf62007271e3b9a91e1c8e29bdece8402bb22012d96571850d65da" Sep 18 17:33:57.573849 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:57.573827 2560 scope.go:117] "RemoveContainer" containerID="47cd29a393cf62007271e3b9a91e1c8e29bdece8402bb22012d96571850d65da" Sep 18 17:33:57.574469 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:57.574439 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47cd29a393cf62007271e3b9a91e1c8e29bdece8402bb22012d96571850d65da\": container with ID starting with 47cd29a393cf62007271e3b9a91e1c8e29bdece8402bb22012d96571850d65da not found: ID does not exist" containerID="47cd29a393cf62007271e3b9a91e1c8e29bdece8402bb22012d96571850d65da" Sep 18 17:33:57.574575 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:57.574481 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47cd29a393cf62007271e3b9a91e1c8e29bdece8402bb22012d96571850d65da"} err="failed to get container status \"47cd29a393cf62007271e3b9a91e1c8e29bdece8402bb22012d96571850d65da\": rpc error: code = NotFound desc = could not find container \"47cd29a393cf62007271e3b9a91e1c8e29bdece8402bb22012d96571850d65da\": container with ID starting with 47cd29a393cf62007271e3b9a91e1c8e29bdece8402bb22012d96571850d65da not found: ID does not exist" Sep 18 17:33:57.587970 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:57.587933 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" podStartSLOduration=2.587920716 podStartE2EDuration="2.587920716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:33:57.586493324 +0000 UTC m=+1150.839654173" watchObservedRunningTime="2026-09-18 17:33:57.587920716 +0000 UTC m=+1150.841081598" Sep 18 17:33:57.693120 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:57.693090 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:57.955194 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:57.955162 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7"] Sep 18 17:33:57.955433 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:57.955411 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerName="istio-proxy" containerID="cri-o://246af98795d4b87cb576437788f3f339ebccff43fe32f3eb11425f3d201bd0f2" gracePeriod=30 Sep 18 17:33:57.963280 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:57.963257 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:58.333254 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.333170 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:58.337921 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.337898 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:58.490426 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.490395 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:58.570061 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.570036 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:58.570997 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.570981 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:33:58.740206 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.740178 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-9c8020" Sep 18 17:33:58.760232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.760198 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9"] Sep 18 17:33:58.763209 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.763190 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.765652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.765633 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-eeab6e-openshift-default-dockercfg-478nz\"" Sep 18 17:33:58.778913 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.778887 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9"] Sep 18 17:33:58.788370 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.788346 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knf4z\" (UniqueName: \"kubernetes.io/projected/0e3c9ee3-a806-413f-9186-6c8695f0c5fa-kube-api-access-knf4z\") pod \"0e3c9ee3-a806-413f-9186-6c8695f0c5fa\" (UID: \"0e3c9ee3-a806-413f-9186-6c8695f0c5fa\") " Sep 18 17:33:58.788582 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.788561 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-podinfo\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.788648 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.788634 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-envoy\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.788699 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.788687 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-credential-socket\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.788794 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.788775 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-workload-certs\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.788830 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.788812 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istiod-ca-cert\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.788880 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.788850 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-workload-socket\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.788939 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.788919 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-data\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.789040 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.789024 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwzn9\" (UniqueName: \"kubernetes.io/projected/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-kube-api-access-nwzn9\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.789102 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.789075 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-token\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.790578 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.790557 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3c9ee3-a806-413f-9186-6c8695f0c5fa-kube-api-access-knf4z" (OuterVolumeSpecName: "kube-api-access-knf4z") pod "0e3c9ee3-a806-413f-9186-6c8695f0c5fa" (UID: "0e3c9ee3-a806-413f-9186-6c8695f0c5fa"). InnerVolumeSpecName "kube-api-access-knf4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:33:58.801601 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.801570 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:58.866369 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.866273 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e3c9ee3-a806-413f-9186-6c8695f0c5fa" path="/var/lib/kubelet/pods/0e3c9ee3-a806-413f-9186-6c8695f0c5fa/volumes" Sep 18 17:33:58.889206 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.889178 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:58.889681 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.889666 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-envoy\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.889731 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.889695 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-credential-socket\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.889840 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.889819 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-workload-certs\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.889902 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.889885 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istiod-ca-cert\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.889953 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.889934 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-workload-socket\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.890008 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.889996 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-data\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.890128 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.890107 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwzn9\" (UniqueName: \"kubernetes.io/projected/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-kube-api-access-nwzn9\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.890188 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.890157 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-token\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.890188 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.890165 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-workload-certs\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.890310 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.890224 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-podinfo\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.890367 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.890353 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-data\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.890429 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.890117 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-credential-socket\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.890482 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.890444 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-workload-socket\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.890637 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.890619 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knf4z\" (UniqueName: \"kubernetes.io/projected/0e3c9ee3-a806-413f-9186-6c8695f0c5fa-kube-api-access-knf4z\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:33:58.890765 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.890739 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istiod-ca-cert\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.892555 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.892535 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-envoy\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.892877 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.892856 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-podinfo\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.899553 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.899526 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-token\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.900615 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.900594 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwzn9\" (UniqueName: \"kubernetes.io/projected/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-kube-api-access-nwzn9\") pod \"e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:58.994895 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:58.994848 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:59.075480 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.075447 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:33:59.354008 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.353831 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9"] Sep 18 17:33:59.357084 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:33:59.357050 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33da0e8_b61f_4bca_aa2a_afc1cdb5628c.slice/crio-3dde24efc2bef95e36899be7802e03575e2891ec6dd172284b60e63ac0e70498 WatchSource:0}: Error finding container 3dde24efc2bef95e36899be7802e03575e2891ec6dd172284b60e63ac0e70498: Status 404 returned error can't find the container with id 3dde24efc2bef95e36899be7802e03575e2891ec6dd172284b60e63ac0e70498 Sep 18 17:33:59.362574 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.362534 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:59.362667 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.362612 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:59.362667 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.362643 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:33:59.561661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.561625 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:59.580622 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.580585 2560 generic.go:356] "Generic (PLEG): container finished" podID="0e3c9ee3-a806-413f-9186-6c8695f0c5fa" containerID="688d4a77161936abd32d772c376574a5ed9b2306dab23081de4eed9cd3ca2c2f" exitCode=137 Sep 18 17:33:59.581016 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.580627 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7723-9c8020" Sep 18 17:33:59.581016 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.580688 2560 scope.go:117] "RemoveContainer" containerID="688d4a77161936abd32d772c376574a5ed9b2306dab23081de4eed9cd3ca2c2f" Sep 18 17:33:59.582973 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.582940 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" event={"ID":"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c","Type":"ContainerStarted","Data":"10daff737d529a3a15178c24d0c763fbc94277cda82b2a83ca4c131da55ad2ea"} Sep 18 17:33:59.583081 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.583009 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" event={"ID":"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c","Type":"ContainerStarted","Data":"3dde24efc2bef95e36899be7802e03575e2891ec6dd172284b60e63ac0e70498"} Sep 18 17:33:59.590721 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.590526 2560 scope.go:117] "RemoveContainer" containerID="688d4a77161936abd32d772c376574a5ed9b2306dab23081de4eed9cd3ca2c2f" Sep 18 17:33:59.590915 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:33:59.590892 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688d4a77161936abd32d772c376574a5ed9b2306dab23081de4eed9cd3ca2c2f\": container with ID starting with 688d4a77161936abd32d772c376574a5ed9b2306dab23081de4eed9cd3ca2c2f not found: ID does not exist" containerID="688d4a77161936abd32d772c376574a5ed9b2306dab23081de4eed9cd3ca2c2f" Sep 18 17:33:59.590960 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.590929 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688d4a77161936abd32d772c376574a5ed9b2306dab23081de4eed9cd3ca2c2f"} err="failed to get container status \"688d4a77161936abd32d772c376574a5ed9b2306dab23081de4eed9cd3ca2c2f\": rpc error: code = NotFound desc = could not find container \"688d4a77161936abd32d772c376574a5ed9b2306dab23081de4eed9cd3ca2c2f\": container with ID starting with 688d4a77161936abd32d772c376574a5ed9b2306dab23081de4eed9cd3ca2c2f not found: ID does not exist" Sep 18 17:33:59.611000 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.610937 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" podStartSLOduration=1.610915385 podStartE2EDuration="1.610915385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:33:59.607468166 +0000 UTC m=+1152.860629004" watchObservedRunningTime="2026-09-18 17:33:59.610915385 +0000 UTC m=+1152.864076216" Sep 18 17:33:59.661221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.661193 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:33:59.906948 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:33:59.906871 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:00.075837 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:00.075754 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:34:00.081015 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:00.080981 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:34:00.487268 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:00.487236 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:00.590617 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:00.590590 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:34:00.591593 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:00.591578 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:34:00.928196 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:00.928167 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:01.574756 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:01.574726 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:01.675704 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:01.675671 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:01.788984 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:01.788949 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:01.892588 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:01.892488 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:03.223407 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.223339 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:34:03.251948 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.251905 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:03.335607 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.335573 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-workload-certs\") pod \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " Sep 18 17:34:03.335607 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.335609 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcb9n\" (UniqueName: \"kubernetes.io/projected/6d34c7af-3bbd-4d5a-bec3-b470994af0da-kube-api-access-vcb9n\") pod \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " Sep 18 17:34:03.335869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.335666 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-data\") pod \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " Sep 18 17:34:03.335869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.335693 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-podinfo\") pod \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " Sep 18 17:34:03.335869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.335820 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istiod-ca-cert\") pod \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " Sep 18 17:34:03.335869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.335855 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-token\") pod \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " Sep 18 17:34:03.336128 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.335882 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-workload-socket\") pod \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " Sep 18 17:34:03.336128 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.335917 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6d34c7af-3bbd-4d5a-bec3-b470994af0da" (UID: "6d34c7af-3bbd-4d5a-bec3-b470994af0da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:34:03.336128 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.335926 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-envoy\") pod \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " Sep 18 17:34:03.336128 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.336002 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-credential-socket\") pod \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\" (UID: \"6d34c7af-3bbd-4d5a-bec3-b470994af0da\") " Sep 18 17:34:03.336128 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.336018 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-data" (OuterVolumeSpecName: "istio-data") pod "6d34c7af-3bbd-4d5a-bec3-b470994af0da" (UID: "6d34c7af-3bbd-4d5a-bec3-b470994af0da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:34:03.336410 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.336208 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6d34c7af-3bbd-4d5a-bec3-b470994af0da" (UID: "6d34c7af-3bbd-4d5a-bec3-b470994af0da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:34:03.336410 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.336232 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6d34c7af-3bbd-4d5a-bec3-b470994af0da" (UID: "6d34c7af-3bbd-4d5a-bec3-b470994af0da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:34:03.336518 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.336413 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6d34c7af-3bbd-4d5a-bec3-b470994af0da" (UID: "6d34c7af-3bbd-4d5a-bec3-b470994af0da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:34:03.336575 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.336553 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:03.336626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.336573 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:03.336626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.336586 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:03.336626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.336599 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:03.336626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.336607 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:03.338192 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.338168 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-token" (OuterVolumeSpecName: "istio-token") pod "6d34c7af-3bbd-4d5a-bec3-b470994af0da" (UID: "6d34c7af-3bbd-4d5a-bec3-b470994af0da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:34:03.338349 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.338200 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d34c7af-3bbd-4d5a-bec3-b470994af0da-kube-api-access-vcb9n" (OuterVolumeSpecName: "kube-api-access-vcb9n") pod "6d34c7af-3bbd-4d5a-bec3-b470994af0da" (UID: "6d34c7af-3bbd-4d5a-bec3-b470994af0da"). InnerVolumeSpecName "kube-api-access-vcb9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:34:03.338434 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.338415 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6d34c7af-3bbd-4d5a-bec3-b470994af0da" (UID: "6d34c7af-3bbd-4d5a-bec3-b470994af0da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:34:03.338487 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.338422 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6d34c7af-3bbd-4d5a-bec3-b470994af0da" (UID: "6d34c7af-3bbd-4d5a-bec3-b470994af0da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:34:03.437779 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.437746 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:03.437779 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.437774 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:03.437779 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.437783 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d34c7af-3bbd-4d5a-bec3-b470994af0da-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:03.438014 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.437792 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcb9n\" (UniqueName: \"kubernetes.io/projected/6d34c7af-3bbd-4d5a-bec3-b470994af0da-kube-api-access-vcb9n\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:03.609993 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.609905 2560 generic.go:356] "Generic (PLEG): container finished" podID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerID="246af98795d4b87cb576437788f3f339ebccff43fe32f3eb11425f3d201bd0f2" exitCode=0 Sep 18 17:34:03.609993 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.609956 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" Sep 18 17:34:03.610185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.609957 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" event={"ID":"6d34c7af-3bbd-4d5a-bec3-b470994af0da","Type":"ContainerDied","Data":"246af98795d4b87cb576437788f3f339ebccff43fe32f3eb11425f3d201bd0f2"} Sep 18 17:34:03.610185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.610114 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7" event={"ID":"6d34c7af-3bbd-4d5a-bec3-b470994af0da","Type":"ContainerDied","Data":"cb115eda343c22b279e73144851924c48cdaec73a06273c02a245b0380170d40"} Sep 18 17:34:03.610185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.610147 2560 scope.go:117] "RemoveContainer" containerID="246af98795d4b87cb576437788f3f339ebccff43fe32f3eb11425f3d201bd0f2" Sep 18 17:34:03.620045 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.620029 2560 scope.go:117] "RemoveContainer" containerID="54fae24d3592f8f51db53690294b32e2f2d4824cb3168653e1c61acceb458ca8" Sep 18 17:34:03.629240 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.629223 2560 scope.go:117] "RemoveContainer" containerID="246af98795d4b87cb576437788f3f339ebccff43fe32f3eb11425f3d201bd0f2" Sep 18 17:34:03.629543 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:34:03.629519 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246af98795d4b87cb576437788f3f339ebccff43fe32f3eb11425f3d201bd0f2\": container with ID starting with 246af98795d4b87cb576437788f3f339ebccff43fe32f3eb11425f3d201bd0f2 not found: ID does not exist" containerID="246af98795d4b87cb576437788f3f339ebccff43fe32f3eb11425f3d201bd0f2" Sep 18 17:34:03.629607 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.629553 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246af98795d4b87cb576437788f3f339ebccff43fe32f3eb11425f3d201bd0f2"} err="failed to get container status \"246af98795d4b87cb576437788f3f339ebccff43fe32f3eb11425f3d201bd0f2\": rpc error: code = NotFound desc = could not find container \"246af98795d4b87cb576437788f3f339ebccff43fe32f3eb11425f3d201bd0f2\": container with ID starting with 246af98795d4b87cb576437788f3f339ebccff43fe32f3eb11425f3d201bd0f2 not found: ID does not exist" Sep 18 17:34:03.629607 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.629571 2560 scope.go:117] "RemoveContainer" containerID="54fae24d3592f8f51db53690294b32e2f2d4824cb3168653e1c61acceb458ca8" Sep 18 17:34:03.629831 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:34:03.629816 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54fae24d3592f8f51db53690294b32e2f2d4824cb3168653e1c61acceb458ca8\": container with ID starting with 54fae24d3592f8f51db53690294b32e2f2d4824cb3168653e1c61acceb458ca8 not found: ID does not exist" containerID="54fae24d3592f8f51db53690294b32e2f2d4824cb3168653e1c61acceb458ca8" Sep 18 17:34:03.629885 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.629833 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54fae24d3592f8f51db53690294b32e2f2d4824cb3168653e1c61acceb458ca8"} err="failed to get container status \"54fae24d3592f8f51db53690294b32e2f2d4824cb3168653e1c61acceb458ca8\": rpc error: code = NotFound desc = could not find container \"54fae24d3592f8f51db53690294b32e2f2d4824cb3168653e1c61acceb458ca8\": container with ID starting with 54fae24d3592f8f51db53690294b32e2f2d4824cb3168653e1c61acceb458ca8 not found: ID does not exist" Sep 18 17:34:03.633171 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.633150 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7"] Sep 18 17:34:03.641728 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.641707 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2aa65a-openshift-default-6759db9d4-8m5z7"] Sep 18 17:34:03.914859 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.914827 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp"] Sep 18 17:34:03.915376 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.915361 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerName="istio-proxy" Sep 18 17:34:03.915465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.915378 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerName="istio-proxy" Sep 18 17:34:03.915465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.915457 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerName="istio-proxy" Sep 18 17:34:03.918638 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.918618 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:03.921259 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.921184 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 17:34:03.921475 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.921414 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sjtq4\"" Sep 18 17:34:03.921661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.921633 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 18 17:34:03.922058 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.922034 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b68d6946-serving-cert\"" Sep 18 17:34:03.922139 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.922056 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 18 17:34:03.928611 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.928588 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp"] Sep 18 17:34:03.941673 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.941646 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f635eaf-169b-44e0-83ab-717872abba74-openshift-service-ca\") pod \"maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:03.941816 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.941681 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-maas-api-tls\") pod \"maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:03.941816 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.941769 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9xhf\" (UniqueName: \"kubernetes.io/projected/0f635eaf-169b-44e0-83ab-717872abba74-kube-api-access-w9xhf\") pod \"maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:03.941905 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.941871 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-metrics-certs\") pod \"maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:03.944502 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:03.944474 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:04.042626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.042595 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f635eaf-169b-44e0-83ab-717872abba74-openshift-service-ca\") pod \"maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:04.042834 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.042632 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-maas-api-tls\") pod \"maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:04.042834 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.042679 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9xhf\" (UniqueName: \"kubernetes.io/projected/0f635eaf-169b-44e0-83ab-717872abba74-kube-api-access-w9xhf\") pod \"maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:04.042834 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.042743 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-metrics-certs\") pod \"maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:04.043001 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:34:04.042827 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-serving-cert: secret "maas-api-e2e-mt-b68d6946-serving-cert" not found Sep 18 17:34:04.043001 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:34:04.042904 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-maas-api-tls podName:0f635eaf-169b-44e0-83ab-717872abba74 nodeName:}" failed. No retries permitted until 2026-09-18 17:34:04.54288204 +0000 UTC m=+1157.796042851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-maas-api-tls") pod "maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" (UID: "0f635eaf-169b-44e0-83ab-717872abba74") : secret "maas-api-e2e-mt-b68d6946-serving-cert" not found Sep 18 17:34:04.043260 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.043243 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f635eaf-169b-44e0-83ab-717872abba74-openshift-service-ca\") pod \"maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:04.045319 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.045272 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:04.045645 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.045624 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-metrics-certs\") pod \"maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:04.052038 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.052019 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9xhf\" (UniqueName: \"kubernetes.io/projected/0f635eaf-169b-44e0-83ab-717872abba74-kube-api-access-w9xhf\") pod \"maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:04.235872 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.235792 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:04.400237 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.400206 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:04.484232 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.484196 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:04.549542 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.549444 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-maas-api-tls\") pod \"maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:04.552285 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.552256 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-maas-api-tls\") pod \"maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:04.798682 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.798498 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv"] Sep 18 17:34:04.799239 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.799222 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerName="istio-proxy" Sep 18 17:34:04.799239 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.799241 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerName="istio-proxy" Sep 18 17:34:04.799384 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.799352 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" containerName="istio-proxy" Sep 18 17:34:04.801657 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.801609 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:04.804167 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.804144 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-eeab6e-serving-cert\"" Sep 18 17:34:04.812319 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.812282 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv"] Sep 18 17:34:04.832579 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.832557 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:04.833536 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.833507 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:04.852991 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.852958 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:04.853103 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.853087 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ppvl\" (UniqueName: \"kubernetes.io/projected/972da268-f047-46c4-a566-ad813b1a0687-kube-api-access-7ppvl\") pod \"maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:04.853168 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.853129 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972da268-f047-46c4-a566-ad813b1a0687-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:04.853239 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.853221 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:04.867698 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.867672 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d34c7af-3bbd-4d5a-bec3-b470994af0da" path="/var/lib/kubelet/pods/6d34c7af-3bbd-4d5a-bec3-b470994af0da/volumes" Sep 18 17:34:04.933236 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.933205 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:04.954321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.954264 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:04.954494 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.954405 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ppvl\" (UniqueName: \"kubernetes.io/projected/972da268-f047-46c4-a566-ad813b1a0687-kube-api-access-7ppvl\") pod \"maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:04.954494 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.954447 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972da268-f047-46c4-a566-ad813b1a0687-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:04.954615 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:34:04.954490 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-serving-cert: secret "maas-api-e2e-shared-a-w3-eeab6e-serving-cert" not found Sep 18 17:34:04.954615 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.954543 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:04.954615 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:34:04.954560 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-maas-api-tls podName:972da268-f047-46c4-a566-ad813b1a0687 nodeName:}" failed. No retries permitted until 2026-09-18 17:34:05.454538627 +0000 UTC m=+1158.707699438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-maas-api-tls") pod "maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" (UID: "972da268-f047-46c4-a566-ad813b1a0687") : secret "maas-api-e2e-shared-a-w3-eeab6e-serving-cert" not found Sep 18 17:34:04.955216 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.955195 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972da268-f047-46c4-a566-ad813b1a0687-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:04.956990 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.956973 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:04.964860 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:04.964841 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ppvl\" (UniqueName: \"kubernetes.io/projected/972da268-f047-46c4-a566-ad813b1a0687-kube-api-access-7ppvl\") pod \"maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:05.072377 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:05.072282 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:05.111104 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:05.111082 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp"] Sep 18 17:34:05.113460 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:34:05.113436 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f635eaf_169b_44e0_83ab_717872abba74.slice/crio-70d65285acf9f8e515a0e6d4f093a23deb85d0058ac8e6753da9753de956473d WatchSource:0}: Error finding container 70d65285acf9f8e515a0e6d4f093a23deb85d0058ac8e6753da9753de956473d: Status 404 returned error can't find the container with id 70d65285acf9f8e515a0e6d4f093a23deb85d0058ac8e6753da9753de956473d Sep 18 17:34:05.296377 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:05.296348 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:05.423394 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:05.423363 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:05.466610 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:05.466092 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:05.482407 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:05.482344 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:05.639021 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:05.638982 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" event={"ID":"0f635eaf-169b-44e0-83ab-717872abba74","Type":"ContainerStarted","Data":"70d65285acf9f8e515a0e6d4f093a23deb85d0058ac8e6753da9753de956473d"} Sep 18 17:34:05.652321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:05.652279 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb"] Sep 18 17:34:05.652577 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:05.652500 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" podUID="08b9c3a5-2f82-4869-942c-30a5701716b1" containerName="istio-proxy" containerID="cri-o://3054d640b33f6e9848451d24a7db6017b2204ca4503db1511919294824cfc167" gracePeriod=30 Sep 18 17:34:05.672179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:05.672146 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:05.714719 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:05.714640 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:05.830876 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:05.830841 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:05.931459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:05.931426 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:06.023024 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:06.023002 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv"] Sep 18 17:34:06.024803 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:06.024773 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:06.025350 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:34:06.025328 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972da268_f047_46c4_a566_ad813b1a0687.slice/crio-61015f3846de31ebf8c0136556135dd63b72aad590651d5ab6b7bc5d58a83dd7 WatchSource:0}: Error finding container 61015f3846de31ebf8c0136556135dd63b72aad590651d5ab6b7bc5d58a83dd7: Status 404 returned error can't find the container with id 61015f3846de31ebf8c0136556135dd63b72aad590651d5ab6b7bc5d58a83dd7 Sep 18 17:34:06.647608 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:06.647579 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" event={"ID":"0f635eaf-169b-44e0-83ab-717872abba74","Type":"ContainerStarted","Data":"2b1c1224da209963cc52c73065a26fef4c7d275f9712a9869e5c196f938bf5e6"} Sep 18 17:34:06.648062 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:06.647681 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:06.649362 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:06.649287 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" event={"ID":"972da268-f047-46c4-a566-ad813b1a0687","Type":"ContainerStarted","Data":"d59ad26ffccf46c2b1fd4f224d28b2eefe74406507076e5d4c262810e7b2fb0e"} Sep 18 17:34:06.649469 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:06.649382 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" event={"ID":"972da268-f047-46c4-a566-ad813b1a0687","Type":"ContainerStarted","Data":"61015f3846de31ebf8c0136556135dd63b72aad590651d5ab6b7bc5d58a83dd7"} Sep 18 17:34:06.649469 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:06.649402 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:06.703954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:06.703907 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" podStartSLOduration=2.64512564 podStartE2EDuration="3.70389113s" totalImagesPullingTime="1.05876549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:34:06.700566534 +0000 UTC m=+1159.953727507" watchObservedRunningTime="2026-09-18 17:34:06.70389113 +0000 UTC m=+1159.957051946" Sep 18 17:34:06.752027 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:06.751973 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" podStartSLOduration=2.438196774 podStartE2EDuration="2.751958263s" totalImagesPullingTime="313.761489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:34:06.748313678 +0000 UTC m=+1160.001474511" watchObservedRunningTime="2026-09-18 17:34:06.751958263 +0000 UTC m=+1160.005119070" Sep 18 17:34:06.841350 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:06.841245 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:07.644606 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:07.644570 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:08.422135 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:08.422101 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:09.115814 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:09.115784 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:09.846361 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:09.846327 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:10.917250 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:10.917191 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:34:11.021623 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.021586 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:11.030637 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.030603 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b9c3a5-2f82-4869-942c-30a5701716b1-istiod-ca-cert\") pod \"08b9c3a5-2f82-4869-942c-30a5701716b1\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " Sep 18 17:34:11.030775 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.030663 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-workload-certs\") pod \"08b9c3a5-2f82-4869-942c-30a5701716b1\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " Sep 18 17:34:11.030775 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.030704 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-credential-socket\") pod \"08b9c3a5-2f82-4869-942c-30a5701716b1\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " Sep 18 17:34:11.030775 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.030759 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-podinfo\") pod \"08b9c3a5-2f82-4869-942c-30a5701716b1\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " Sep 18 17:34:11.030958 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.030822 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-workload-socket\") pod \"08b9c3a5-2f82-4869-942c-30a5701716b1\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " Sep 18 17:34:11.030958 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.030872 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-token\") pod \"08b9c3a5-2f82-4869-942c-30a5701716b1\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " Sep 18 17:34:11.030958 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.030916 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-data\") pod \"08b9c3a5-2f82-4869-942c-30a5701716b1\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " Sep 18 17:34:11.030958 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.030950 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4txwk\" (UniqueName: \"kubernetes.io/projected/08b9c3a5-2f82-4869-942c-30a5701716b1-kube-api-access-4txwk\") pod \"08b9c3a5-2f82-4869-942c-30a5701716b1\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " Sep 18 17:34:11.031178 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.031001 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-envoy\") pod \"08b9c3a5-2f82-4869-942c-30a5701716b1\" (UID: \"08b9c3a5-2f82-4869-942c-30a5701716b1\") " Sep 18 17:34:11.031266 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.031020 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b9c3a5-2f82-4869-942c-30a5701716b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08b9c3a5-2f82-4869-942c-30a5701716b1" (UID: "08b9c3a5-2f82-4869-942c-30a5701716b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:34:11.031347 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.031018 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08b9c3a5-2f82-4869-942c-30a5701716b1" (UID: "08b9c3a5-2f82-4869-942c-30a5701716b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:34:11.031347 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.031169 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "08b9c3a5-2f82-4869-942c-30a5701716b1" (UID: "08b9c3a5-2f82-4869-942c-30a5701716b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:34:11.031347 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.031199 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08b9c3a5-2f82-4869-942c-30a5701716b1" (UID: "08b9c3a5-2f82-4869-942c-30a5701716b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:34:11.031610 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.031359 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08b9c3a5-2f82-4869-942c-30a5701716b1" (UID: "08b9c3a5-2f82-4869-942c-30a5701716b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:34:11.031737 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.031717 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:11.031823 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.031741 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:11.031823 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.031755 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08b9c3a5-2f82-4869-942c-30a5701716b1-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:11.031823 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.031768 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:11.031823 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.031782 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:11.033268 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.033244 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b9c3a5-2f82-4869-942c-30a5701716b1-kube-api-access-4txwk" (OuterVolumeSpecName: "kube-api-access-4txwk") pod "08b9c3a5-2f82-4869-942c-30a5701716b1" (UID: "08b9c3a5-2f82-4869-942c-30a5701716b1"). InnerVolumeSpecName "kube-api-access-4txwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:34:11.033386 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.033352 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "08b9c3a5-2f82-4869-942c-30a5701716b1" (UID: "08b9c3a5-2f82-4869-942c-30a5701716b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:34:11.033600 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.033580 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08b9c3a5-2f82-4869-942c-30a5701716b1" (UID: "08b9c3a5-2f82-4869-942c-30a5701716b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:34:11.033671 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.033654 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08b9c3a5-2f82-4869-942c-30a5701716b1" (UID: "08b9c3a5-2f82-4869-942c-30a5701716b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:34:11.132841 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.132801 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:11.132841 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.132837 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4txwk\" (UniqueName: \"kubernetes.io/projected/08b9c3a5-2f82-4869-942c-30a5701716b1-kube-api-access-4txwk\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:11.132841 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.132849 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:11.133090 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.132862 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08b9c3a5-2f82-4869-942c-30a5701716b1-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:34:11.687888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.687852 2560 generic.go:356] "Generic (PLEG): container finished" podID="08b9c3a5-2f82-4869-942c-30a5701716b1" containerID="3054d640b33f6e9848451d24a7db6017b2204ca4503db1511919294824cfc167" exitCode=0 Sep 18 17:34:11.688042 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.687903 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" Sep 18 17:34:11.688042 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.687939 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" event={"ID":"08b9c3a5-2f82-4869-942c-30a5701716b1","Type":"ContainerDied","Data":"3054d640b33f6e9848451d24a7db6017b2204ca4503db1511919294824cfc167"} Sep 18 17:34:11.688042 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.687999 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb" event={"ID":"08b9c3a5-2f82-4869-942c-30a5701716b1","Type":"ContainerDied","Data":"4236a8be737315555ab896ed2059bb66f57120e529ff200a6ebbbb79dfc60f24"} Sep 18 17:34:11.688042 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.688021 2560 scope.go:117] "RemoveContainer" containerID="3054d640b33f6e9848451d24a7db6017b2204ca4503db1511919294824cfc167" Sep 18 17:34:11.698009 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.697992 2560 scope.go:117] "RemoveContainer" containerID="3054d640b33f6e9848451d24a7db6017b2204ca4503db1511919294824cfc167" Sep 18 17:34:11.698275 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:34:11.698255 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3054d640b33f6e9848451d24a7db6017b2204ca4503db1511919294824cfc167\": container with ID starting with 3054d640b33f6e9848451d24a7db6017b2204ca4503db1511919294824cfc167 not found: ID does not exist" containerID="3054d640b33f6e9848451d24a7db6017b2204ca4503db1511919294824cfc167" Sep 18 17:34:11.698391 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.698282 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3054d640b33f6e9848451d24a7db6017b2204ca4503db1511919294824cfc167"} err="failed to get container status \"3054d640b33f6e9848451d24a7db6017b2204ca4503db1511919294824cfc167\": rpc error: code = NotFound desc = could not find container \"3054d640b33f6e9848451d24a7db6017b2204ca4503db1511919294824cfc167\": container with ID starting with 3054d640b33f6e9848451d24a7db6017b2204ca4503db1511919294824cfc167 not found: ID does not exist" Sep 18 17:34:11.711330 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.711285 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb"] Sep 18 17:34:11.720378 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.720354 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-231bbc-openshift-default-86688568c4-k5wpb"] Sep 18 17:34:11.914315 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:11.913542 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:34:12.673751 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:12.673720 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:34:12.674263 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:12.674110 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:34:12.866079 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:12.866046 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b9c3a5-2f82-4869-942c-30a5701716b1" path="/var/lib/kubelet/pods/08b9c3a5-2f82-4869-942c-30a5701716b1/volumes" Sep 18 17:34:22.796049 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:22.796015 2560 generic.go:356] "Generic (PLEG): container finished" podID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerID="f575c6113ed8a54210da49e946667546336027c4be06bf2626a880b13219ed5e" exitCode=0 Sep 18 17:34:22.796557 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:22.796084 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" event={"ID":"58f4236b-5856-469a-bd03-76bfe6c2b3e1","Type":"ContainerDied","Data":"f575c6113ed8a54210da49e946667546336027c4be06bf2626a880b13219ed5e"} Sep 18 17:34:22.796621 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:22.796557 2560 scope.go:117] "RemoveContainer" containerID="f575c6113ed8a54210da49e946667546336027c4be06bf2626a880b13219ed5e" Sep 18 17:34:22.798054 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:22.798015 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:34:22.798130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:22.798116 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:34:22.798181 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:22.798165 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:34:23.804477 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:23.804447 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" event={"ID":"58f4236b-5856-469a-bd03-76bfe6c2b3e1","Type":"ContainerStarted","Data":"1b649e332b8ef7e0fefb397d7a09a31c167185369f318d6c031f192649de8331"} Sep 18 17:34:24.333653 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:24.333615 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:34:24.334936 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:24.334911 2560 patch_prober.go:28] interesting pod/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 18 17:34:24.335057 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:24.334962 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" podUID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 18 17:34:24.811745 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:24.811712 2560 generic.go:356] "Generic (PLEG): container finished" podID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerID="10daff737d529a3a15178c24d0c763fbc94277cda82b2a83ca4c131da55ad2ea" exitCode=0 Sep 18 17:34:24.812219 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:24.811785 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" event={"ID":"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c","Type":"ContainerDied","Data":"10daff737d529a3a15178c24d0c763fbc94277cda82b2a83ca4c131da55ad2ea"} Sep 18 17:34:24.812410 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:24.812394 2560 scope.go:117] "RemoveContainer" containerID="10daff737d529a3a15178c24d0c763fbc94277cda82b2a83ca4c131da55ad2ea" Sep 18 17:34:24.813531 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:24.813503 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:34:24.813615 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:24.813565 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:34:24.813615 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:24.813597 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:34:24.885522 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:24.885490 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:25.333258 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:25.333223 2560 patch_prober.go:28] interesting pod/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 18 17:34:25.333457 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:25.333274 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" podUID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 18 17:34:25.819772 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:25.819741 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" event={"ID":"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c","Type":"ContainerStarted","Data":"6fbc910e67e3197727180876972cb5182787b2de24c55caac63a09234b4b4c65"} Sep 18 17:34:26.076104 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:26.075949 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:34:26.077826 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:26.077795 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 18 17:34:26.078058 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:26.078030 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 18 17:34:26.263465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:26.263194 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:26.333677 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:26.333596 2560 patch_prober.go:28] interesting pod/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 18 17:34:26.333677 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:26.333649 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" podUID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 18 17:34:27.076836 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:27.076805 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 18 17:34:27.077445 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:27.077395 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 18 17:34:27.130430 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:27.130386 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:27.333614 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:27.333529 2560 patch_prober.go:28] interesting pod/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 18 17:34:27.333614 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:27.333589 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" podUID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 18 17:34:27.933501 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:27.933469 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:28.075824 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:28.075708 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 18 17:34:28.075824 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:28.075793 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 18 17:34:28.334087 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:28.334007 2560 patch_prober.go:28] interesting pod/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 18 17:34:28.334555 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:28.334530 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" podUID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 18 17:34:28.570229 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:28.570177 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:34:28.849971 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:28.849915 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:29.076585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:29.076540 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 18 17:34:29.076769 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:29.076608 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 18 17:34:29.417786 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:29.417748 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:34:29.419477 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:29.419449 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:34:29.559259 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:29.559226 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:29.654844 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:29.654813 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:30.076766 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:30.076730 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 18 17:34:30.076930 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:30.076786 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 18 17:34:30.287569 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:30.287533 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:30.591015 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:30.590977 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:34:31.076659 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:31.076625 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 18 17:34:31.076852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:31.076687 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 18 17:34:31.217971 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:31.217931 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:31.513233 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:31.513196 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:31.825836 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:31.825754 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:32.073473 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.073442 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:32.081496 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.081423 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:34:32.082869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.082849 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:34:32.191046 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.191017 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66"] Sep 18 17:34:32.191615 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.191599 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b9c3a5-2f82-4869-942c-30a5701716b1" containerName="istio-proxy" Sep 18 17:34:32.191615 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.191616 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b9c3a5-2f82-4869-942c-30a5701716b1" containerName="istio-proxy" Sep 18 17:34:32.191748 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.191712 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b9c3a5-2f82-4869-942c-30a5701716b1" containerName="istio-proxy" Sep 18 17:34:32.194842 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.194823 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.198989 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.198970 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-808470-openshift-default-dockercfg-2jrlp\"" Sep 18 17:34:32.211435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.211408 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66"] Sep 18 17:34:32.214347 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.214326 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:32.288952 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.288905 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-credential-socket\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.288952 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.288956 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-podinfo\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.289233 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.289051 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-envoy\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.289233 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.289079 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istiod-ca-cert\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.289233 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.289115 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-workload-socket\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.289233 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.289134 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-data\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.289233 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.289184 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-token\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.289233 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.289210 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-workload-certs\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.289593 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.289311 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9js9\" (UniqueName: \"kubernetes.io/projected/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-kube-api-access-w9js9\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.390539 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.390451 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istiod-ca-cert\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.390539 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.390499 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-workload-socket\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.390539 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.390516 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-data\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.390818 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.390560 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-token\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.390818 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.390581 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-workload-certs\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.390818 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.390658 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9js9\" (UniqueName: \"kubernetes.io/projected/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-kube-api-access-w9js9\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.390818 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.390703 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-credential-socket\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.390818 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.390727 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-podinfo\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.391086 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.390833 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-envoy\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.391086 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.390919 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-workload-socket\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.391198 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.391112 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-credential-socket\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.391198 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.391144 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-data\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.391198 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.391163 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-workload-certs\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.391335 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.391319 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istiod-ca-cert\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.393188 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.393170 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-envoy\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.393343 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.393326 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-podinfo\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.402840 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.402813 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-token\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.402971 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.402924 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9js9\" (UniqueName: \"kubernetes.io/projected/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-kube-api-access-w9js9\") pod \"e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.506936 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.506905 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:32.816805 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.816770 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66"] Sep 18 17:34:32.819019 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:34:32.818989 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f3fb3b_7f0e_4bc5_a5ce_c88804fbdad9.slice/crio-330bc4176c2b35fe539464904262fa7b151c3c02fa327f2226dd8c79d0292404 WatchSource:0}: Error finding container 330bc4176c2b35fe539464904262fa7b151c3c02fa327f2226dd8c79d0292404: Status 404 returned error can't find the container with id 330bc4176c2b35fe539464904262fa7b151c3c02fa327f2226dd8c79d0292404 Sep 18 17:34:32.821448 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.821408 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:34:32.821529 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.821501 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:34:32.821573 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.821545 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:34:32.869671 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.869641 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" event={"ID":"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9","Type":"ContainerStarted","Data":"330bc4176c2b35fe539464904262fa7b151c3c02fa327f2226dd8c79d0292404"} Sep 18 17:34:32.971518 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:32.971487 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:33.154154 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:33.154114 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:33.877419 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:33.877380 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" event={"ID":"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9","Type":"ContainerStarted","Data":"2c3dd83cc2c5bbf4b8da151bce6862f4bf5cf8a4b678e3336f695637b917f178"} Sep 18 17:34:33.903525 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:33.903476 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" podStartSLOduration=1.903460097 podStartE2EDuration="1.903460097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:34:33.90142369 +0000 UTC m=+1187.154584506" watchObservedRunningTime="2026-09-18 17:34:33.903460097 +0000 UTC m=+1187.156620959" Sep 18 17:34:34.508066 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:34.508025 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:34.513742 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:34.513716 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:34.883204 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:34.883112 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:34.884158 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:34.884138 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:34:35.159023 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:35.158990 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:35.780753 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:35.780717 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:35.872789 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:35.872759 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:36.063765 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:36.063678 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:37.673609 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:37.673572 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:39.532116 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:39.532085 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:40.423630 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:40.423600 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:41.470462 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:41.470429 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:42.433594 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:42.433557 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:43.305342 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.305310 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp"] Sep 18 17:34:43.308625 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.308600 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:43.311951 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.311931 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-808470-serving-cert\"" Sep 18 17:34:43.320575 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.320538 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp"] Sep 18 17:34:43.339907 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.339859 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:43.414729 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.414693 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft4pv\" (UniqueName: \"kubernetes.io/projected/587f3ae3-56fa-4ece-8580-f9787cdc0db4-kube-api-access-ft4pv\") pod \"maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:43.414960 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.414802 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587f3ae3-56fa-4ece-8580-f9787cdc0db4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:43.414960 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.414879 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:43.414960 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.414917 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:43.445685 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.445648 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:43.516382 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.516345 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft4pv\" (UniqueName: \"kubernetes.io/projected/587f3ae3-56fa-4ece-8580-f9787cdc0db4-kube-api-access-ft4pv\") pod \"maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:43.516568 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.516407 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587f3ae3-56fa-4ece-8580-f9787cdc0db4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:43.516568 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.516435 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:43.516568 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.516454 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:43.516700 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:34:43.516568 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-serving-cert: secret "maas-api-e2e-shared-b-w3-808470-serving-cert" not found Sep 18 17:34:43.516700 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:34:43.516655 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-maas-api-tls podName:587f3ae3-56fa-4ece-8580-f9787cdc0db4 nodeName:}" failed. No retries permitted until 2026-09-18 17:34:44.016635384 +0000 UTC m=+1197.269796179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-maas-api-tls") pod "maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" (UID: "587f3ae3-56fa-4ece-8580-f9787cdc0db4") : secret "maas-api-e2e-shared-b-w3-808470-serving-cert" not found Sep 18 17:34:43.517196 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.517177 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587f3ae3-56fa-4ece-8580-f9787cdc0db4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:43.519040 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.519016 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:43.525953 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.525924 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft4pv\" (UniqueName: \"kubernetes.io/projected/587f3ae3-56fa-4ece-8580-f9787cdc0db4-kube-api-access-ft4pv\") pod \"maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:43.861271 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.861234 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:43.971130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:43.971091 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:44.021965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:44.021928 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:44.024355 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:44.024328 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:44.058545 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:44.058509 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:44.222645 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:44.222605 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:44.505436 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:44.505409 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp"] Sep 18 17:34:44.506903 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:34:44.506878 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587f3ae3_56fa_4ece_8580_f9787cdc0db4.slice/crio-d37804ae228cc699692238ef10e1dfef4369619058fa5ebadd4b802b9a9b57f6 WatchSource:0}: Error finding container d37804ae228cc699692238ef10e1dfef4369619058fa5ebadd4b802b9a9b57f6: Status 404 returned error can't find the container with id d37804ae228cc699692238ef10e1dfef4369619058fa5ebadd4b802b9a9b57f6 Sep 18 17:34:44.956335 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:44.955749 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" event={"ID":"587f3ae3-56fa-4ece-8580-f9787cdc0db4","Type":"ContainerStarted","Data":"8e46c9e3632ec2a158a2d1401a62c8ddf61ced4a769594148f1d10e82c93a452"} Sep 18 17:34:44.956335 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:44.955860 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" event={"ID":"587f3ae3-56fa-4ece-8580-f9787cdc0db4","Type":"ContainerStarted","Data":"d37804ae228cc699692238ef10e1dfef4369619058fa5ebadd4b802b9a9b57f6"} Sep 18 17:34:44.956335 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:44.955891 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:44.980027 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:44.979972 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" podStartSLOduration=1.610691632 podStartE2EDuration="1.979952683s" totalImagesPullingTime="369.261051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:34:44.977264209 +0000 UTC m=+1198.230425026" watchObservedRunningTime="2026-09-18 17:34:44.979952683 +0000 UTC m=+1198.233113501" Sep 18 17:34:47.622488 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:47.622458 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:47.833632 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:47.833596 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:48.404699 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:48.404660 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:34:48.901468 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:48.901442 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:34:48.904310 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:48.904275 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:34:48.907600 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:48.907580 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:34:48.910549 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:48.910534 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:34:50.966872 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:50.966840 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:34:57.310504 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:34:57.310465 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:35:04.432737 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:04.432704 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:05.508368 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:05.508287 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:05.844011 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:05.843978 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:05.945816 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:05.945777 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:06.662067 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:06.662033 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:07.541756 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:07.541721 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:08.293244 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:08.293216 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:08.961709 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:08.961652 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:09.117149 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:09.117115 2560 generic.go:356] "Generic (PLEG): container finished" podID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerID="2c3dd83cc2c5bbf4b8da151bce6862f4bf5cf8a4b678e3336f695637b917f178" exitCode=0 Sep 18 17:35:09.117355 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:09.117189 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" event={"ID":"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9","Type":"ContainerDied","Data":"2c3dd83cc2c5bbf4b8da151bce6862f4bf5cf8a4b678e3336f695637b917f178"} Sep 18 17:35:09.117663 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:09.117645 2560 scope.go:117] "RemoveContainer" containerID="2c3dd83cc2c5bbf4b8da151bce6862f4bf5cf8a4b678e3336f695637b917f178" Sep 18 17:35:09.118842 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:09.118812 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:35:09.118903 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:09.118889 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:35:09.118946 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:09.118923 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:35:09.741558 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:09.741514 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:10.125779 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:10.125696 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" event={"ID":"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9","Type":"ContainerStarted","Data":"deedd1d11afe08fcbe960f6a0708d837d7d9f15c0a0f5a8abba30fdf4321c14e"} Sep 18 17:35:10.432576 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:10.432541 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:10.508093 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:10.508061 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:35:10.508280 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:10.508261 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 18 17:35:10.508354 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:10.508335 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 18 17:35:11.336986 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:11.336951 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:11.507667 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:11.507629 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 18 17:35:11.507853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:11.507704 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 18 17:35:12.125190 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:12.125159 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:12.196790 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:12.196759 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:12.507538 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:12.507507 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 18 17:35:12.507907 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:12.507557 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 18 17:35:13.507897 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:13.507862 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 18 17:35:13.508279 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:13.507927 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 18 17:35:14.508140 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:14.508104 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 18 17:35:14.508565 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:14.508159 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 18 17:35:15.507349 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:15.507285 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 18 17:35:15.507539 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:15.507376 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 18 17:35:16.511498 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:16.511467 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:35:16.511938 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:16.511891 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:35:16.512489 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:16.512473 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:35:34.205251 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:34.205168 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:35:37.337515 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:37.337483 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:43.657776 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:43.657742 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:35:49.027571 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:49.027526 2560 scope.go:117] "RemoveContainer" containerID="eec1f0979419f96d85b63d49733b08e45a9893af91278527df3f9b900b9043f5" Sep 18 17:35:55.132827 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:35:55.132794 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:06.697108 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:06.697071 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:09.717707 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:09.717667 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:36:31.018629 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:31.018595 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:36.690661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:36.690623 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:37.961652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:37.961618 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:38.795659 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:38.795627 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:39.701095 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:39.701057 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:40.526132 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:40.526068 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:40.594276 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:40.594245 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:41.298756 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:41.298726 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:42.194882 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:42.194851 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:42.864182 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:42.864152 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:43.679829 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:43.679793 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:49.055935 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:49.055904 2560 scope.go:117] "RemoveContainer" containerID="b3b83cf985b8a24204dd3097a231ca54cb433486100888fb3b79590464cc770f" Sep 18 17:36:51.983628 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:51.983588 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp"] Sep 18 17:36:51.984103 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:51.983917 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" podUID="0f635eaf-169b-44e0-83ab-717872abba74" containerName="maas-api" containerID="cri-o://2b1c1224da209963cc52c73065a26fef4c7d275f9712a9869e5c196f938bf5e6" gracePeriod=30 Sep 18 17:36:52.013677 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.013642 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:52.093041 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.093004 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:52.193247 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.193207 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:52.256153 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.256075 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:36:52.281017 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.280980 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:52.343350 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.343284 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:52.352502 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.352472 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f635eaf-169b-44e0-83ab-717872abba74-openshift-service-ca\") pod \"0f635eaf-169b-44e0-83ab-717872abba74\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " Sep 18 17:36:52.352699 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.352528 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-metrics-certs\") pod \"0f635eaf-169b-44e0-83ab-717872abba74\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " Sep 18 17:36:52.352699 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.352582 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9xhf\" (UniqueName: \"kubernetes.io/projected/0f635eaf-169b-44e0-83ab-717872abba74-kube-api-access-w9xhf\") pod \"0f635eaf-169b-44e0-83ab-717872abba74\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " Sep 18 17:36:52.352699 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.352654 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-maas-api-tls\") pod \"0f635eaf-169b-44e0-83ab-717872abba74\" (UID: \"0f635eaf-169b-44e0-83ab-717872abba74\") " Sep 18 17:36:52.352956 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.352920 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f635eaf-169b-44e0-83ab-717872abba74-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f635eaf-169b-44e0-83ab-717872abba74" (UID: "0f635eaf-169b-44e0-83ab-717872abba74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:36:52.353158 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.353130 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f635eaf-169b-44e0-83ab-717872abba74-openshift-service-ca\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:36:52.355389 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.355357 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f635eaf-169b-44e0-83ab-717872abba74" (UID: "0f635eaf-169b-44e0-83ab-717872abba74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:36:52.355498 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.355452 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f635eaf-169b-44e0-83ab-717872abba74-kube-api-access-w9xhf" (OuterVolumeSpecName: "kube-api-access-w9xhf") pod "0f635eaf-169b-44e0-83ab-717872abba74" (UID: "0f635eaf-169b-44e0-83ab-717872abba74"). InnerVolumeSpecName "kube-api-access-w9xhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:36:52.355498 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.355485 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0f635eaf-169b-44e0-83ab-717872abba74" (UID: "0f635eaf-169b-44e0-83ab-717872abba74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:36:52.454557 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.454526 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-metrics-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:36:52.454557 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.454551 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9xhf\" (UniqueName: \"kubernetes.io/projected/0f635eaf-169b-44e0-83ab-717872abba74-kube-api-access-w9xhf\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:36:52.454557 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.454560 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f635eaf-169b-44e0-83ab-717872abba74-maas-api-tls\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:36:52.759740 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.759704 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:52.765792 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.765760 2560 generic.go:356] "Generic (PLEG): container finished" podID="0f635eaf-169b-44e0-83ab-717872abba74" containerID="2b1c1224da209963cc52c73065a26fef4c7d275f9712a9869e5c196f938bf5e6" exitCode=0 Sep 18 17:36:52.765981 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.765813 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" Sep 18 17:36:52.765981 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.765841 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" event={"ID":"0f635eaf-169b-44e0-83ab-717872abba74","Type":"ContainerDied","Data":"2b1c1224da209963cc52c73065a26fef4c7d275f9712a9869e5c196f938bf5e6"} Sep 18 17:36:52.765981 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.765894 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp" event={"ID":"0f635eaf-169b-44e0-83ab-717872abba74","Type":"ContainerDied","Data":"70d65285acf9f8e515a0e6d4f093a23deb85d0058ac8e6753da9753de956473d"} Sep 18 17:36:52.765981 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.765916 2560 scope.go:117] "RemoveContainer" containerID="2b1c1224da209963cc52c73065a26fef4c7d275f9712a9869e5c196f938bf5e6" Sep 18 17:36:52.776233 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.776214 2560 scope.go:117] "RemoveContainer" containerID="2b1c1224da209963cc52c73065a26fef4c7d275f9712a9869e5c196f938bf5e6" Sep 18 17:36:52.776569 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:36:52.776545 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1c1224da209963cc52c73065a26fef4c7d275f9712a9869e5c196f938bf5e6\": container with ID starting with 2b1c1224da209963cc52c73065a26fef4c7d275f9712a9869e5c196f938bf5e6 not found: ID does not exist" containerID="2b1c1224da209963cc52c73065a26fef4c7d275f9712a9869e5c196f938bf5e6" Sep 18 17:36:52.776636 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.776579 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1c1224da209963cc52c73065a26fef4c7d275f9712a9869e5c196f938bf5e6"} err="failed to get container status \"2b1c1224da209963cc52c73065a26fef4c7d275f9712a9869e5c196f938bf5e6\": rpc error: code = NotFound desc = could not find container \"2b1c1224da209963cc52c73065a26fef4c7d275f9712a9869e5c196f938bf5e6\": container with ID starting with 2b1c1224da209963cc52c73065a26fef4c7d275f9712a9869e5c196f938bf5e6 not found: ID does not exist" Sep 18 17:36:52.789082 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.789046 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp"] Sep 18 17:36:52.795176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.795151 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b68d6946-6b6cf59648-vgpvp"] Sep 18 17:36:52.864855 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:52.864826 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f635eaf-169b-44e0-83ab-717872abba74" path="/var/lib/kubelet/pods/0f635eaf-169b-44e0-83ab-717872abba74/volumes" Sep 18 17:36:53.474937 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:53.474903 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:54.399084 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:54.399054 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:55.065790 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:55.065763 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:55.323829 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:55.323750 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:55.405724 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:55.405693 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:36:55.719046 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:55.719010 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:56.440086 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:56.440025 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:57.140828 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:57.140791 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:36:58.608796 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:36:58.608762 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:00.361687 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:00.361602 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:11.051188 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:11.051154 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:24.233308 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:24.233271 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:27.853695 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:27.853659 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:37:35.616176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:35.616130 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:37.391164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:37.391124 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:37.992330 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:37.992269 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:43.832487 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:43.832455 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:49.852625 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:49.852572 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:49.955481 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:49.955445 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:50.070463 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:50.070427 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k"] Sep 18 17:37:50.070707 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:50.070683 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" podUID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerName="istio-proxy" containerID="cri-o://1b649e332b8ef7e0fefb397d7a09a31c167185369f318d6c031f192649de8331" gracePeriod=30 Sep 18 17:37:50.141581 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:50.141497 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:51.317735 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:51.317705 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:51.459249 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:51.459217 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:51.561181 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:51.561150 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:52.136814 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:52.136761 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:52.254234 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:52.254208 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:52.377216 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:52.377186 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:53.711468 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:53.711436 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:54.825060 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:54.825006 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:37:55.314548 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.314519 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:37:55.364130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.364056 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppzkz\" (UniqueName: \"kubernetes.io/projected/58f4236b-5856-469a-bd03-76bfe6c2b3e1-kube-api-access-ppzkz\") pod \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " Sep 18 17:37:55.364130 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.364103 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-podinfo\") pod \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " Sep 18 17:37:55.364354 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.364139 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-data\") pod \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " Sep 18 17:37:55.364354 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.364201 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-workload-certs\") pod \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " Sep 18 17:37:55.364354 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.364223 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-workload-socket\") pod \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " Sep 18 17:37:55.364354 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.364266 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-envoy\") pod \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " Sep 18 17:37:55.364354 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.364280 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-token\") pod \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " Sep 18 17:37:55.364354 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.364337 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istiod-ca-cert\") pod \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " Sep 18 17:37:55.364668 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.364379 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-credential-socket\") pod \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\" (UID: \"58f4236b-5856-469a-bd03-76bfe6c2b3e1\") " Sep 18 17:37:55.364668 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.364625 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-data" (OuterVolumeSpecName: "istio-data") pod "58f4236b-5856-469a-bd03-76bfe6c2b3e1" (UID: "58f4236b-5856-469a-bd03-76bfe6c2b3e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:37:55.364789 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.364731 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "58f4236b-5856-469a-bd03-76bfe6c2b3e1" (UID: "58f4236b-5856-469a-bd03-76bfe6c2b3e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:37:55.364850 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.364798 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:37:55.364850 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.364841 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "58f4236b-5856-469a-bd03-76bfe6c2b3e1" (UID: "58f4236b-5856-469a-bd03-76bfe6c2b3e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:37:55.365049 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.365003 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "58f4236b-5856-469a-bd03-76bfe6c2b3e1" (UID: "58f4236b-5856-469a-bd03-76bfe6c2b3e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:37:55.365145 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.365052 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "58f4236b-5856-469a-bd03-76bfe6c2b3e1" (UID: "58f4236b-5856-469a-bd03-76bfe6c2b3e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:37:55.366834 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.366791 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-token" (OuterVolumeSpecName: "istio-token") pod "58f4236b-5856-469a-bd03-76bfe6c2b3e1" (UID: "58f4236b-5856-469a-bd03-76bfe6c2b3e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:37:55.366944 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.366865 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "58f4236b-5856-469a-bd03-76bfe6c2b3e1" (UID: "58f4236b-5856-469a-bd03-76bfe6c2b3e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:37:55.366944 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.366909 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "58f4236b-5856-469a-bd03-76bfe6c2b3e1" (UID: "58f4236b-5856-469a-bd03-76bfe6c2b3e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:37:55.367025 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.366962 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f4236b-5856-469a-bd03-76bfe6c2b3e1-kube-api-access-ppzkz" (OuterVolumeSpecName: "kube-api-access-ppzkz") pod "58f4236b-5856-469a-bd03-76bfe6c2b3e1" (UID: "58f4236b-5856-469a-bd03-76bfe6c2b3e1"). InnerVolumeSpecName "kube-api-access-ppzkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:37:55.465757 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.465714 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppzkz\" (UniqueName: \"kubernetes.io/projected/58f4236b-5856-469a-bd03-76bfe6c2b3e1-kube-api-access-ppzkz\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:37:55.465757 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.465750 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:37:55.465757 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.465763 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:37:55.466046 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.465775 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:37:55.466046 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.465785 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:37:55.466046 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.465796 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:37:55.466046 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.465806 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f4236b-5856-469a-bd03-76bfe6c2b3e1-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:37:55.466046 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:55.465815 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f4236b-5856-469a-bd03-76bfe6c2b3e1-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:37:56.152782 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:56.152747 2560 generic.go:356] "Generic (PLEG): container finished" podID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerID="1b649e332b8ef7e0fefb397d7a09a31c167185369f318d6c031f192649de8331" exitCode=0 Sep 18 17:37:56.153244 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:56.152793 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" Sep 18 17:37:56.153244 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:56.152827 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" event={"ID":"58f4236b-5856-469a-bd03-76bfe6c2b3e1","Type":"ContainerDied","Data":"1b649e332b8ef7e0fefb397d7a09a31c167185369f318d6c031f192649de8331"} Sep 18 17:37:56.153244 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:56.152888 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k" event={"ID":"58f4236b-5856-469a-bd03-76bfe6c2b3e1","Type":"ContainerDied","Data":"1393bf48205b1eb816f7ef228b26586712dff40fcf469426c74b8d91134f7223"} Sep 18 17:37:56.153244 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:56.152918 2560 scope.go:117] "RemoveContainer" containerID="1b649e332b8ef7e0fefb397d7a09a31c167185369f318d6c031f192649de8331" Sep 18 17:37:56.163478 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:56.163459 2560 scope.go:117] "RemoveContainer" containerID="f575c6113ed8a54210da49e946667546336027c4be06bf2626a880b13219ed5e" Sep 18 17:37:56.174502 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:56.174479 2560 scope.go:117] "RemoveContainer" containerID="1b649e332b8ef7e0fefb397d7a09a31c167185369f318d6c031f192649de8331" Sep 18 17:37:56.174805 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:37:56.174786 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b649e332b8ef7e0fefb397d7a09a31c167185369f318d6c031f192649de8331\": container with ID starting with 1b649e332b8ef7e0fefb397d7a09a31c167185369f318d6c031f192649de8331 not found: ID does not exist" containerID="1b649e332b8ef7e0fefb397d7a09a31c167185369f318d6c031f192649de8331" Sep 18 17:37:56.174870 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:56.174812 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b649e332b8ef7e0fefb397d7a09a31c167185369f318d6c031f192649de8331"} err="failed to get container status \"1b649e332b8ef7e0fefb397d7a09a31c167185369f318d6c031f192649de8331\": rpc error: code = NotFound desc = could not find container \"1b649e332b8ef7e0fefb397d7a09a31c167185369f318d6c031f192649de8331\": container with ID starting with 1b649e332b8ef7e0fefb397d7a09a31c167185369f318d6c031f192649de8331 not found: ID does not exist" Sep 18 17:37:56.174870 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:56.174839 2560 scope.go:117] "RemoveContainer" containerID="f575c6113ed8a54210da49e946667546336027c4be06bf2626a880b13219ed5e" Sep 18 17:37:56.175110 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:37:56.175091 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f575c6113ed8a54210da49e946667546336027c4be06bf2626a880b13219ed5e\": container with ID starting with f575c6113ed8a54210da49e946667546336027c4be06bf2626a880b13219ed5e not found: ID does not exist" containerID="f575c6113ed8a54210da49e946667546336027c4be06bf2626a880b13219ed5e" Sep 18 17:37:56.175165 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:56.175116 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f575c6113ed8a54210da49e946667546336027c4be06bf2626a880b13219ed5e"} err="failed to get container status \"f575c6113ed8a54210da49e946667546336027c4be06bf2626a880b13219ed5e\": rpc error: code = NotFound desc = could not find container \"f575c6113ed8a54210da49e946667546336027c4be06bf2626a880b13219ed5e\": container with ID starting with f575c6113ed8a54210da49e946667546336027c4be06bf2626a880b13219ed5e not found: ID does not exist" Sep 18 17:37:56.176829 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:56.176806 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k"] Sep 18 17:37:56.180023 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:56.180000 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b68d6946-openshift-default-574bff878d-5j86k"] Sep 18 17:37:56.866631 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:56.866598 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" path="/var/lib/kubelet/pods/58f4236b-5856-469a-bd03-76bfe6c2b3e1/volumes" Sep 18 17:37:59.925973 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:37:59.925940 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:00.027455 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:00.027422 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:00.609519 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:00.609486 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:02.331795 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:02.331759 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:38:06.576948 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:06.576918 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:06.690587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:06.690552 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:06.788262 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:06.788232 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:09.419484 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.419450 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk"] Sep 18 17:38:09.419972 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.419959 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f635eaf-169b-44e0-83ab-717872abba74" containerName="maas-api" Sep 18 17:38:09.419972 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.419973 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f635eaf-169b-44e0-83ab-717872abba74" containerName="maas-api" Sep 18 17:38:09.420070 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.419982 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerName="istio-proxy" Sep 18 17:38:09.420070 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.419989 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerName="istio-proxy" Sep 18 17:38:09.420070 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.420004 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerName="istio-proxy" Sep 18 17:38:09.420070 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.420010 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerName="istio-proxy" Sep 18 17:38:09.420209 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.420094 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerName="istio-proxy" Sep 18 17:38:09.420209 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.420102 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f635eaf-169b-44e0-83ab-717872abba74" containerName="maas-api" Sep 18 17:38:09.420209 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.420110 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="58f4236b-5856-469a-bd03-76bfe6c2b3e1" containerName="istio-proxy" Sep 18 17:38:09.423257 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.423240 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.427028 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.427007 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5b6d69\"/\"default-dockercfg-r86nm\"" Sep 18 17:38:09.427168 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.427008 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5b6d69\"/\"openshift-service-ca.crt\"" Sep 18 17:38:09.428218 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.428198 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-5b6d69\"/\"kube-root-ca.crt\"" Sep 18 17:38:09.428348 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.428197 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-5b6d69\"/\"distinct-2-w4-5b6d69-kserve-self-signed-certs\"" Sep 18 17:38:09.435611 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.435564 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk"] Sep 18 17:38:09.517203 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.517168 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-home\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.517203 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.517218 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstlf\" (UniqueName: \"kubernetes.io/projected/22816025-feac-448a-a289-063138c3d64c-kube-api-access-pstlf\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.517443 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.517340 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-tmp-dir\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.517443 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.517384 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-model-cache\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.517443 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.517412 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22816025-feac-448a-a289-063138c3d64c-tls-certs\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.517548 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.517475 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-dshm\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.519581 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.519557 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:09.619096 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.619055 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-home\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.619096 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.619109 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstlf\" (UniqueName: \"kubernetes.io/projected/22816025-feac-448a-a289-063138c3d64c-kube-api-access-pstlf\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.619390 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.619182 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-tmp-dir\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.619390 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.619223 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-model-cache\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.619390 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.619256 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22816025-feac-448a-a289-063138c3d64c-tls-certs\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.619390 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.619364 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-dshm\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.619632 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.619548 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-home\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.619632 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.619599 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-tmp-dir\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.619740 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.619664 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-model-cache\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.621558 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.621531 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-dshm\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.621839 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.621818 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22816025-feac-448a-a289-063138c3d64c-tls-certs\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.629217 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.629186 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstlf\" (UniqueName: \"kubernetes.io/projected/22816025-feac-448a-a289-063138c3d64c-kube-api-access-pstlf\") pod \"distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:09.753029 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:09.752927 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:10.047348 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:10.044881 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk"] Sep 18 17:38:10.236731 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:10.236694 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" event={"ID":"22816025-feac-448a-a289-063138c3d64c","Type":"ContainerStarted","Data":"3d3d47967f49d266620f660c32b203cf69c5f49229fdfc7ee7df5b2e67954144"} Sep 18 17:38:10.236731 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:10.236741 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" event={"ID":"22816025-feac-448a-a289-063138c3d64c","Type":"ContainerStarted","Data":"78845c44f5ed780a02f632328d59a7a314cc206409c5fb7e157aca176a6f684f"} Sep 18 17:38:10.261129 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:10.261070 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" podStartSLOduration=1.261043109 podStartE2EDuration="1.261043109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:38:10.258162023 +0000 UTC m=+1403.511322853" watchObservedRunningTime="2026-09-18 17:38:10.261043109 +0000 UTC m=+1403.514203926" Sep 18 17:38:11.903976 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:11.903943 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:17.315220 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:17.315180 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:17.562192 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:17.562158 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:19.753407 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:19.753369 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:19.753812 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:19.753429 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:19.766359 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:19.766285 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:20.311235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:20.311197 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:38:36.703845 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:36.703764 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:38.461394 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:38.461359 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:39.405100 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:39.405054 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:38:44.970716 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:44.970679 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:45.065857 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.065821 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:45.276841 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.276763 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:45.377087 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.377051 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:45.583328 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.583240 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4"] Sep 18 17:38:45.586912 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.586897 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.589809 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.589792 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9c0254-openshift-default-dockercfg-txbw9\"" Sep 18 17:38:45.610349 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.610311 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4"] Sep 18 17:38:45.614710 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.614685 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-envoy\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.614837 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.614725 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-workload-certs\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.614837 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.614809 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk674\" (UniqueName: \"kubernetes.io/projected/aac5040f-d169-4dcb-8a78-4b4e243b87a9-kube-api-access-pk674\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.614940 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.614856 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-podinfo\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.614940 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.614885 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-workload-socket\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.615008 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.614949 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-token\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.615008 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.614980 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-credential-socket\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.615008 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.614999 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istiod-ca-cert\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.615116 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.615051 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-data\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.646862 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.646830 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:45.716873 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.716802 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-envoy\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.717179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.716893 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-workload-certs\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.717398 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.717367 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk674\" (UniqueName: \"kubernetes.io/projected/aac5040f-d169-4dcb-8a78-4b4e243b87a9-kube-api-access-pk674\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.717488 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.717469 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-workload-certs\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.717549 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.717493 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-podinfo\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.718386 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.718359 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-workload-socket\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.718526 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.718457 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-token\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.718526 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.718495 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-credential-socket\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.718526 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.718518 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istiod-ca-cert\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.719162 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.718763 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-data\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.719162 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.719032 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-credential-socket\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.719162 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.719102 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-data\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.719590 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.719525 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-workload-socket\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.720235 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.720212 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istiod-ca-cert\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.720422 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.720402 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-envoy\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.721561 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.721538 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-podinfo\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.727435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.727415 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk674\" (UniqueName: \"kubernetes.io/projected/aac5040f-d169-4dcb-8a78-4b4e243b87a9-kube-api-access-pk674\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.732197 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.732180 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-token\") pod \"e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.898132 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.898106 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:45.920845 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:45.920815 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:46.182474 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:46.182444 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4"] Sep 18 17:38:46.183440 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:38:46.183413 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac5040f_d169_4dcb_8a78_4b4e243b87a9.slice/crio-6724899ba7e5491802d9cc0bb1097bd4b1c1b97e91ed51c096551038da2e988c WatchSource:0}: Error finding container 6724899ba7e5491802d9cc0bb1097bd4b1c1b97e91ed51c096551038da2e988c: Status 404 returned error can't find the container with id 6724899ba7e5491802d9cc0bb1097bd4b1c1b97e91ed51c096551038da2e988c Sep 18 17:38:46.185646 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:46.185615 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:38:46.185731 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:46.185683 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:38:46.185731 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:46.185713 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:38:46.466576 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:46.466490 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" event={"ID":"aac5040f-d169-4dcb-8a78-4b4e243b87a9","Type":"ContainerStarted","Data":"07b9318e84864063432af940645ad9a4230220f7f0849c434860a75a900950d6"} Sep 18 17:38:46.466576 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:46.466545 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" event={"ID":"aac5040f-d169-4dcb-8a78-4b4e243b87a9","Type":"ContainerStarted","Data":"6724899ba7e5491802d9cc0bb1097bd4b1c1b97e91ed51c096551038da2e988c"} Sep 18 17:38:46.489601 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:46.489553 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" podStartSLOduration=1.489538865 podStartE2EDuration="1.489538865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:38:46.488131602 +0000 UTC m=+1439.741292465" watchObservedRunningTime="2026-09-18 17:38:46.489538865 +0000 UTC m=+1439.742699682" Sep 18 17:38:46.898931 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:46.898907 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:47.713942 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:47.713910 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:47.903983 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:47.903952 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:48.436696 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:48.436661 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:48.478885 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:48.478857 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:48.479897 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:48.479881 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:38:48.509443 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:48.509398 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:49.258513 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:49.258482 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:50.039740 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:50.039710 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:50.779186 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:50.779155 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:51.511878 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:51.511847 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:52.432519 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:52.432485 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:53.102908 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.102879 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p"] Sep 18 17:38:53.108001 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.107981 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.110530 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.110510 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9c0254-serving-cert\"" Sep 18 17:38:53.122429 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.122405 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p"] Sep 18 17:38:53.166182 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.166156 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:53.196367 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.196329 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5cf23bb-7230-4fa3-a127-195cec5093b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.196519 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.196432 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5cf23bb-7230-4fa3-a127-195cec5093b3-maas-api-tls\") pod \"maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.196519 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.196458 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5cf23bb-7230-4fa3-a127-195cec5093b3-metrics-certs\") pod \"maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.196519 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.196508 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57jqk\" (UniqueName: \"kubernetes.io/projected/a5cf23bb-7230-4fa3-a127-195cec5093b3-kube-api-access-57jqk\") pod \"maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.279288 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.279237 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:53.298063 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.298037 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5cf23bb-7230-4fa3-a127-195cec5093b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.298213 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.298095 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5cf23bb-7230-4fa3-a127-195cec5093b3-maas-api-tls\") pod \"maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.298213 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.298113 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5cf23bb-7230-4fa3-a127-195cec5093b3-metrics-certs\") pod \"maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.298213 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.298147 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jqk\" (UniqueName: \"kubernetes.io/projected/a5cf23bb-7230-4fa3-a127-195cec5093b3-kube-api-access-57jqk\") pod \"maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.298820 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.298795 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5cf23bb-7230-4fa3-a127-195cec5093b3-openshift-service-ca\") pod \"maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.300724 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.300701 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5cf23bb-7230-4fa3-a127-195cec5093b3-metrics-certs\") pod \"maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.300808 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.300709 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5cf23bb-7230-4fa3-a127-195cec5093b3-maas-api-tls\") pod \"maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.305866 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.305842 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57jqk\" (UniqueName: \"kubernetes.io/projected/a5cf23bb-7230-4fa3-a127-195cec5093b3-kube-api-access-57jqk\") pod \"maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.418966 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.418933 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:53.601121 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.601087 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:53.695385 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.695352 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p"] Sep 18 17:38:53.696151 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:38:53.696124 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5cf23bb_7230_4fa3_a127_195cec5093b3.slice/crio-2c44a2974ad6b6797d02108139184a8559c17caa53916c76656c52e395b117ae WatchSource:0}: Error finding container 2c44a2974ad6b6797d02108139184a8559c17caa53916c76656c52e395b117ae: Status 404 returned error can't find the container with id 2c44a2974ad6b6797d02108139184a8559c17caa53916c76656c52e395b117ae Sep 18 17:38:53.696262 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.696229 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:38:53.697745 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:53.697727 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 17:38:54.518668 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:54.518621 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" event={"ID":"a5cf23bb-7230-4fa3-a127-195cec5093b3","Type":"ContainerStarted","Data":"300a0b7b68f5b025454f8488a2b46eab861fe5361d612fbb3554bccdda324356"} Sep 18 17:38:54.519140 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:54.518681 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:38:54.519140 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:54.518701 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" event={"ID":"a5cf23bb-7230-4fa3-a127-195cec5093b3","Type":"ContainerStarted","Data":"2c44a2974ad6b6797d02108139184a8559c17caa53916c76656c52e395b117ae"} Sep 18 17:38:54.538251 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:38:54.538198 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" podStartSLOduration=1.102222407 podStartE2EDuration="1.538180757s" totalImagesPullingTime="435.95835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:38:54.535987759 +0000 UTC m=+1447.789148590" watchObservedRunningTime="2026-09-18 17:38:54.538180757 +0000 UTC m=+1447.791341573" Sep 18 17:39:00.533227 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:00.533199 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:39:02.571274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:02.571235 2560 generic.go:356] "Generic (PLEG): container finished" podID="7e88128b-b901-4a7d-905d-53641faa452b" containerID="347ca721606971b4f16d7fd727f0a748e2f20c9fe0a942ed09a1e8a11920d682" exitCode=137 Sep 18 17:39:02.571714 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:02.571323 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" event={"ID":"7e88128b-b901-4a7d-905d-53641faa452b","Type":"ContainerDied","Data":"347ca721606971b4f16d7fd727f0a748e2f20c9fe0a942ed09a1e8a11920d682"} Sep 18 17:39:02.571714 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:02.571702 2560 scope.go:117] "RemoveContainer" containerID="347ca721606971b4f16d7fd727f0a748e2f20c9fe0a942ed09a1e8a11920d682" Sep 18 17:39:03.579857 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:03.579821 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" event={"ID":"7e88128b-b901-4a7d-905d-53641faa452b","Type":"ContainerStarted","Data":"517bc0b2db8c3a611d7bb0b5c95279ae1b2831699f9619df2b020096cb1b1df5"} Sep 18 17:39:03.580265 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:03.580062 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:39:12.069769 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:12.069731 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:12.146002 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:12.145972 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:14.588720 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:14.588689 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:39:16.570606 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:16.570573 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:17.043704 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:17.043665 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:17.753872 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:17.753842 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:18.482223 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:18.482191 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:18.584998 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:18.584966 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:18.971061 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:18.971028 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:19.243185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.243111 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22"] Sep 18 17:39:19.247607 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.247584 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.250196 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.250164 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f4a54f-openshift-default-dockercfg-g54w7\"" Sep 18 17:39:19.260214 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.260192 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22"] Sep 18 17:39:19.270948 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.270928 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:19.280028 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.279991 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-workload-socket\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.280028 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.280027 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-data\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.280239 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.280059 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-token\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.280239 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.280092 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5qh9\" (UniqueName: \"kubernetes.io/projected/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-kube-api-access-f5qh9\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.280239 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.280119 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-credential-socket\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.280239 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.280164 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-envoy\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.280239 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.280228 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-podinfo\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.280717 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.280275 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-workload-certs\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.280717 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.280384 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istiod-ca-cert\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.342068 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.342037 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:19.381412 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.381382 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-envoy\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.381601 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.381439 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-podinfo\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.381601 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.381471 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-workload-certs\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.381601 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.381508 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istiod-ca-cert\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.381601 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.381584 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-workload-socket\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.381601 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.381599 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-data\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.381865 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.381618 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-token\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.381865 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.381640 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5qh9\" (UniqueName: \"kubernetes.io/projected/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-kube-api-access-f5qh9\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.381865 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.381666 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-credential-socket\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.382056 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.381892 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-workload-certs\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.382349 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.382324 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-data\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.382451 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.382345 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-workload-socket\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.382451 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.382402 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-credential-socket\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.382451 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.382434 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istiod-ca-cert\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.383916 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.383899 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-envoy\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.384139 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.384121 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-podinfo\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.391045 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.391026 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-token\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.391370 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.391353 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5qh9\" (UniqueName: \"kubernetes.io/projected/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-kube-api-access-f5qh9\") pod \"e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.561080 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.560981 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:19.622666 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.622464 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:19.842170 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.842143 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22"] Sep 18 17:39:19.843544 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:39:19.843515 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29db4e1_fcc9_45b2_a4d5_cb8f85fc1068.slice/crio-aee57f611163b651f8aac66ee6381008fab758a14c9ed74215d48349af6366b1 WatchSource:0}: Error finding container aee57f611163b651f8aac66ee6381008fab758a14c9ed74215d48349af6366b1: Status 404 returned error can't find the container with id aee57f611163b651f8aac66ee6381008fab758a14c9ed74215d48349af6366b1 Sep 18 17:39:19.846086 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.846051 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:39:19.846184 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.846118 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:39:19.846184 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:19.846147 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:39:20.024574 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:20.024530 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:20.695736 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:20.695697 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" event={"ID":"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068","Type":"ContainerStarted","Data":"92dea9a0271c67642fb62ff5d82dd952ffa013072b5f931792be77f93644128b"} Sep 18 17:39:20.695944 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:20.695782 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" event={"ID":"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068","Type":"ContainerStarted","Data":"aee57f611163b651f8aac66ee6381008fab758a14c9ed74215d48349af6366b1"} Sep 18 17:39:20.721150 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:20.721091 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" podStartSLOduration=1.7210653919999999 podStartE2EDuration="1.721065392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:39:20.719668537 +0000 UTC m=+1473.972829367" watchObservedRunningTime="2026-09-18 17:39:20.721065392 +0000 UTC m=+1473.974226212" Sep 18 17:39:20.839240 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:20.839212 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:21.561498 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:21.561467 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:21.566744 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:21.566718 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:21.614695 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:21.614664 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:21.701469 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:21.701440 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:21.702473 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:21.702454 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:39:22.433706 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:22.433669 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:22.774178 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:22.774091 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:22.863931 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:22.863897 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:23.420015 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:23.419983 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:24.213516 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:24.213482 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:25.064068 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:25.064035 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:25.702137 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:25.702104 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:26.711857 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:26.711807 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:27.228171 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:27.228135 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:27.844938 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:27.844904 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:28.602846 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.602808 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf"] Sep 18 17:39:28.608285 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.608260 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:28.610834 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.610812 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f4a54f-serving-cert\"" Sep 18 17:39:28.615779 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.615750 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf"] Sep 18 17:39:28.633997 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.633968 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:28.687459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.687422 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txfsn\" (UniqueName: \"kubernetes.io/projected/2283838e-e81b-400a-a329-fbbd9b9dc75d-kube-api-access-txfsn\") pod \"maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:28.687613 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.687549 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2283838e-e81b-400a-a329-fbbd9b9dc75d-openshift-service-ca\") pod \"maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:28.687613 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.687598 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2283838e-e81b-400a-a329-fbbd9b9dc75d-metrics-certs\") pod \"maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:28.687709 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.687621 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2283838e-e81b-400a-a329-fbbd9b9dc75d-maas-api-tls\") pod \"maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:28.701895 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.701865 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:28.789028 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.788991 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2283838e-e81b-400a-a329-fbbd9b9dc75d-openshift-service-ca\") pod \"maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:28.789216 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.789043 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2283838e-e81b-400a-a329-fbbd9b9dc75d-metrics-certs\") pod \"maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:28.789216 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.789071 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2283838e-e81b-400a-a329-fbbd9b9dc75d-maas-api-tls\") pod \"maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:28.789333 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.789221 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txfsn\" (UniqueName: \"kubernetes.io/projected/2283838e-e81b-400a-a329-fbbd9b9dc75d-kube-api-access-txfsn\") pod \"maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:28.789816 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.789787 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2283838e-e81b-400a-a329-fbbd9b9dc75d-openshift-service-ca\") pod \"maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:28.791723 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.791696 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2283838e-e81b-400a-a329-fbbd9b9dc75d-maas-api-tls\") pod \"maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:28.791815 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.791733 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2283838e-e81b-400a-a329-fbbd9b9dc75d-metrics-certs\") pod \"maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:28.797620 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.797598 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txfsn\" (UniqueName: \"kubernetes.io/projected/2283838e-e81b-400a-a329-fbbd9b9dc75d-kube-api-access-txfsn\") pod \"maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:28.801703 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.801670 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:28.922365 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:28.922327 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:29.099728 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:29.099697 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:29.193463 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:29.193381 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:29.210360 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:29.210330 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf"] Sep 18 17:39:29.212501 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:39:29.212467 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2283838e_e81b_400a_a329_fbbd9b9dc75d.slice/crio-e63e06d0db1bb626d21a13d6eadfe103d2cd9d4a5cf22226ae9f4066f7fce93b WatchSource:0}: Error finding container e63e06d0db1bb626d21a13d6eadfe103d2cd9d4a5cf22226ae9f4066f7fce93b: Status 404 returned error can't find the container with id e63e06d0db1bb626d21a13d6eadfe103d2cd9d4a5cf22226ae9f4066f7fce93b Sep 18 17:39:29.499952 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:29.499831 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:29.755990 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:29.755896 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" event={"ID":"2283838e-e81b-400a-a329-fbbd9b9dc75d","Type":"ContainerStarted","Data":"8f7b34d691b91dada39d0761eba5267c80dd5bcb1b273aa930cb62c67f1c8797"} Sep 18 17:39:29.755990 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:29.755951 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:29.755990 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:29.755974 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" event={"ID":"2283838e-e81b-400a-a329-fbbd9b9dc75d","Type":"ContainerStarted","Data":"e63e06d0db1bb626d21a13d6eadfe103d2cd9d4a5cf22226ae9f4066f7fce93b"} Sep 18 17:39:29.776212 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:29.776156 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" podStartSLOduration=1.402921729 podStartE2EDuration="1.776136128s" totalImagesPullingTime="373.214399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:39:29.775697638 +0000 UTC m=+1483.028858475" watchObservedRunningTime="2026-09-18 17:39:29.776136128 +0000 UTC m=+1483.029296945" Sep 18 17:39:29.991783 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:29.991730 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:30.202599 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:30.202566 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:31.217025 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:31.216988 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:39:35.772309 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:35.772271 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:39:39.214935 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:39.214896 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:39.623135 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:39.623100 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:40.722043 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:40.722003 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:41.530158 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:41.530126 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:42.308453 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:42.308415 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:42.599082 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:42.598941 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:42.840667 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:42.840632 2560 generic.go:356] "Generic (PLEG): container finished" podID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerID="07b9318e84864063432af940645ad9a4230220f7f0849c434860a75a900950d6" exitCode=0 Sep 18 17:39:42.840850 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:42.840703 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" event={"ID":"aac5040f-d169-4dcb-8a78-4b4e243b87a9","Type":"ContainerDied","Data":"07b9318e84864063432af940645ad9a4230220f7f0849c434860a75a900950d6"} Sep 18 17:39:42.841210 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:42.841197 2560 scope.go:117] "RemoveContainer" containerID="07b9318e84864063432af940645ad9a4230220f7f0849c434860a75a900950d6" Sep 18 17:39:42.842540 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:42.842511 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:39:42.842611 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:42.842578 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:39:42.842648 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:42.842610 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:39:43.176519 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:43.176476 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:43.851750 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:43.851703 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" event={"ID":"aac5040f-d169-4dcb-8a78-4b4e243b87a9","Type":"ContainerStarted","Data":"e84d06e8475019a7532167b02dd82141597decdafe98a5e644fa10260d6821e4"} Sep 18 17:39:43.898702 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:43.898670 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:39:43.899743 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:43.899717 2560 patch_prober.go:28] interesting pod/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 18 17:39:43.899854 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:43.899769 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" podUID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 18 17:39:44.025118 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:44.025078 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:44.293180 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:44.293150 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:44.842359 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:44.842329 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:44.898465 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:44.898427 2560 patch_prober.go:28] interesting pod/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 18 17:39:44.898853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:44.898482 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" podUID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 18 17:39:45.578071 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:45.578036 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:45.898543 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:45.898514 2560 patch_prober.go:28] interesting pod/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 18 17:39:45.898918 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:45.898570 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" podUID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 18 17:39:46.277175 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:46.277043 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:46.898538 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:46.898501 2560 patch_prober.go:28] interesting pod/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 18 17:39:46.898730 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:46.898557 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" podUID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 18 17:39:47.068468 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:47.068432 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:47.874354 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:47.874313 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:47.898757 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:47.898721 2560 patch_prober.go:28] interesting pod/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 18 17:39:47.899236 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:47.898785 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" podUID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 18 17:39:48.479108 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:48.479068 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:39:48.830065 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:48.829979 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:48.955987 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:48.955958 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:39:48.957723 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:48.957690 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:39:48.962696 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:48.962675 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:39:48.964149 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:48.964126 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:39:49.101144 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:49.101047 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:39:49.102707 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:49.102678 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:39:49.106408 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:49.106381 2560 scope.go:117] "RemoveContainer" containerID="d64082a3e897f9b41a0dae8ddaf1c1a1ffee6af1eccacabe1bbad1d2336ee88d" Sep 18 17:39:49.651207 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:49.651173 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:50.492234 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:50.492203 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:51.244819 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:51.244783 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:52.860516 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:52.860480 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:52.960012 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:52.959983 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:53.619414 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:53.619378 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:54.548989 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:54.548957 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:55.816639 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:55.816609 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:59.413821 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.413790 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:59.484808 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.484756 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q"] Sep 18 17:39:59.486597 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.486559 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" podUID="8e3b4dba-f46c-443d-9728-3a3fb1afe129" containerName="payload-processing" containerID="cri-o://8190354c69605fbe6568ec16b7ddea98bd6b8279dc1ef96663267c824ccf3e55" gracePeriod=30 Sep 18 17:39:59.489468 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.489437 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:59.575023 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.574893 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:59.657018 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.656904 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:59.824226 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.823721 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" Sep 18 17:39:59.978743 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.978632 2560 generic.go:356] "Generic (PLEG): container finished" podID="8e3b4dba-f46c-443d-9728-3a3fb1afe129" containerID="8190354c69605fbe6568ec16b7ddea98bd6b8279dc1ef96663267c824ccf3e55" exitCode=0 Sep 18 17:39:59.978743 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.978682 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" Sep 18 17:39:59.978743 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.978722 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" event={"ID":"8e3b4dba-f46c-443d-9728-3a3fb1afe129","Type":"ContainerDied","Data":"8190354c69605fbe6568ec16b7ddea98bd6b8279dc1ef96663267c824ccf3e55"} Sep 18 17:39:59.978964 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.978777 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q" event={"ID":"8e3b4dba-f46c-443d-9728-3a3fb1afe129","Type":"ContainerDied","Data":"1a7f371c2833a4134a8357e8f360519d5922494ec15633e93410cff7210fe088"} Sep 18 17:39:59.978964 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.978809 2560 scope.go:117] "RemoveContainer" containerID="8190354c69605fbe6568ec16b7ddea98bd6b8279dc1ef96663267c824ccf3e55" Sep 18 17:39:59.988345 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.988322 2560 scope.go:117] "RemoveContainer" containerID="8190354c69605fbe6568ec16b7ddea98bd6b8279dc1ef96663267c824ccf3e55" Sep 18 17:39:59.988659 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:39:59.988635 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8190354c69605fbe6568ec16b7ddea98bd6b8279dc1ef96663267c824ccf3e55\": container with ID starting with 8190354c69605fbe6568ec16b7ddea98bd6b8279dc1ef96663267c824ccf3e55 not found: ID does not exist" containerID="8190354c69605fbe6568ec16b7ddea98bd6b8279dc1ef96663267c824ccf3e55" Sep 18 17:39:59.988712 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.988666 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8190354c69605fbe6568ec16b7ddea98bd6b8279dc1ef96663267c824ccf3e55"} err="failed to get container status \"8190354c69605fbe6568ec16b7ddea98bd6b8279dc1ef96663267c824ccf3e55\": rpc error: code = NotFound desc = could not find container \"8190354c69605fbe6568ec16b7ddea98bd6b8279dc1ef96663267c824ccf3e55\": container with ID starting with 8190354c69605fbe6568ec16b7ddea98bd6b8279dc1ef96663267c824ccf3e55 not found: ID does not exist" Sep 18 17:39:59.991172 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.991150 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:39:59.997370 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.997347 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mclht\" (UniqueName: \"kubernetes.io/projected/8e3b4dba-f46c-443d-9728-3a3fb1afe129-kube-api-access-mclht\") pod \"8e3b4dba-f46c-443d-9728-3a3fb1afe129\" (UID: \"8e3b4dba-f46c-443d-9728-3a3fb1afe129\") " Sep 18 17:39:59.997454 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.997443 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3b4dba-f46c-443d-9728-3a3fb1afe129-plugins-config-volume\") pod \"8e3b4dba-f46c-443d-9728-3a3fb1afe129\" (UID: \"8e3b4dba-f46c-443d-9728-3a3fb1afe129\") " Sep 18 17:39:59.997839 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.997817 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3b4dba-f46c-443d-9728-3a3fb1afe129-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e3b4dba-f46c-443d-9728-3a3fb1afe129" (UID: "8e3b4dba-f46c-443d-9728-3a3fb1afe129"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:39:59.999578 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:39:59.999555 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3b4dba-f46c-443d-9728-3a3fb1afe129-kube-api-access-mclht" (OuterVolumeSpecName: "kube-api-access-mclht") pod "8e3b4dba-f46c-443d-9728-3a3fb1afe129" (UID: "8e3b4dba-f46c-443d-9728-3a3fb1afe129"). InnerVolumeSpecName "kube-api-access-mclht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:40:00.098996 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:00.098962 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mclht\" (UniqueName: \"kubernetes.io/projected/8e3b4dba-f46c-443d-9728-3a3fb1afe129-kube-api-access-mclht\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:00.098996 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:00.098993 2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e3b4dba-f46c-443d-9728-3a3fb1afe129-plugins-config-volume\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:00.244767 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:00.244597 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:00.301864 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:00.301827 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q"] Sep 18 17:40:00.305977 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:00.305944 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-0a2e13-66ff4cfccf-xd25q"] Sep 18 17:40:00.875601 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:00.875561 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e3b4dba-f46c-443d-9728-3a3fb1afe129" path="/var/lib/kubelet/pods/8e3b4dba-f46c-443d-9728-3a3fb1afe129/volumes" Sep 18 17:40:00.917690 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:00.917625 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:01.729208 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:01.729173 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:02.533243 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:02.533196 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:03.249616 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:03.249579 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:03.852387 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:03.852353 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p"] Sep 18 17:40:03.853374 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:03.853181 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" podUID="a5cf23bb-7230-4fa3-a127-195cec5093b3" containerName="maas-api" containerID="cri-o://300a0b7b68f5b025454f8488a2b46eab861fe5361d612fbb3554bccdda324356" gracePeriod=30 Sep 18 17:40:03.880243 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:03.878743 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:03.947156 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:03.947117 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:04.007396 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.007361 2560 generic.go:356] "Generic (PLEG): container finished" podID="a5cf23bb-7230-4fa3-a127-195cec5093b3" containerID="300a0b7b68f5b025454f8488a2b46eab861fe5361d612fbb3554bccdda324356" exitCode=0 Sep 18 17:40:04.007587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.007414 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" event={"ID":"a5cf23bb-7230-4fa3-a127-195cec5093b3","Type":"ContainerDied","Data":"300a0b7b68f5b025454f8488a2b46eab861fe5361d612fbb3554bccdda324356"} Sep 18 17:40:04.117649 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.117568 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:40:04.146684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.145531 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5cf23bb-7230-4fa3-a127-195cec5093b3-openshift-service-ca\") pod \"a5cf23bb-7230-4fa3-a127-195cec5093b3\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " Sep 18 17:40:04.146684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.145658 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57jqk\" (UniqueName: \"kubernetes.io/projected/a5cf23bb-7230-4fa3-a127-195cec5093b3-kube-api-access-57jqk\") pod \"a5cf23bb-7230-4fa3-a127-195cec5093b3\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " Sep 18 17:40:04.146684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.145753 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5cf23bb-7230-4fa3-a127-195cec5093b3-metrics-certs\") pod \"a5cf23bb-7230-4fa3-a127-195cec5093b3\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " Sep 18 17:40:04.146684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.145825 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5cf23bb-7230-4fa3-a127-195cec5093b3-maas-api-tls\") pod \"a5cf23bb-7230-4fa3-a127-195cec5093b3\" (UID: \"a5cf23bb-7230-4fa3-a127-195cec5093b3\") " Sep 18 17:40:04.147875 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.147812 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5cf23bb-7230-4fa3-a127-195cec5093b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a5cf23bb-7230-4fa3-a127-195cec5093b3" (UID: "a5cf23bb-7230-4fa3-a127-195cec5093b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:40:04.150090 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.150047 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5cf23bb-7230-4fa3-a127-195cec5093b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a5cf23bb-7230-4fa3-a127-195cec5093b3" (UID: "a5cf23bb-7230-4fa3-a127-195cec5093b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:40:04.150380 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.150347 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5cf23bb-7230-4fa3-a127-195cec5093b3-kube-api-access-57jqk" (OuterVolumeSpecName: "kube-api-access-57jqk") pod "a5cf23bb-7230-4fa3-a127-195cec5093b3" (UID: "a5cf23bb-7230-4fa3-a127-195cec5093b3"). InnerVolumeSpecName "kube-api-access-57jqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:40:04.151646 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.151621 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5cf23bb-7230-4fa3-a127-195cec5093b3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a5cf23bb-7230-4fa3-a127-195cec5093b3" (UID: "a5cf23bb-7230-4fa3-a127-195cec5093b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:40:04.232820 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.232784 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:04.248480 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.247387 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5cf23bb-7230-4fa3-a127-195cec5093b3-metrics-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:04.248480 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.247421 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5cf23bb-7230-4fa3-a127-195cec5093b3-maas-api-tls\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:04.248480 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.247436 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5cf23bb-7230-4fa3-a127-195cec5093b3-openshift-service-ca\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:04.248480 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.247449 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57jqk\" (UniqueName: \"kubernetes.io/projected/a5cf23bb-7230-4fa3-a127-195cec5093b3-kube-api-access-57jqk\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:04.331911 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.331871 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:04.431277 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:04.431227 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:05.015245 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:05.015161 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" Sep 18 17:40:05.015245 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:05.015198 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p" event={"ID":"a5cf23bb-7230-4fa3-a127-195cec5093b3","Type":"ContainerDied","Data":"2c44a2974ad6b6797d02108139184a8559c17caa53916c76656c52e395b117ae"} Sep 18 17:40:05.015802 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:05.015268 2560 scope.go:117] "RemoveContainer" containerID="300a0b7b68f5b025454f8488a2b46eab861fe5361d612fbb3554bccdda324356" Sep 18 17:40:05.017439 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:05.017411 2560 generic.go:356] "Generic (PLEG): container finished" podID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerID="92dea9a0271c67642fb62ff5d82dd952ffa013072b5f931792be77f93644128b" exitCode=0 Sep 18 17:40:05.017573 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:05.017462 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" event={"ID":"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068","Type":"ContainerDied","Data":"92dea9a0271c67642fb62ff5d82dd952ffa013072b5f931792be77f93644128b"} Sep 18 17:40:05.017934 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:05.017913 2560 scope.go:117] "RemoveContainer" containerID="92dea9a0271c67642fb62ff5d82dd952ffa013072b5f931792be77f93644128b" Sep 18 17:40:05.019176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:05.019145 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:40:05.019279 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:05.019213 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:40:05.019279 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:05.019247 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:40:05.116794 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:05.116757 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p"] Sep 18 17:40:05.140184 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:05.140149 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c0254-56d4f6b84f-nw92p"] Sep 18 17:40:05.273184 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:05.273147 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:06.025755 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:06.025724 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" event={"ID":"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068","Type":"ContainerStarted","Data":"376e73792e309f8d19fa640e15b13e3d4e21939a8d2c5eb1aeff06c6d6aab24d"} Sep 18 17:40:06.522049 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:06.522020 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:06.562490 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:06.562433 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:40:06.563761 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:06.563734 2560 patch_prober.go:28] interesting pod/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 18 17:40:06.564738 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:06.563962 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" podUID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 18 17:40:06.702019 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:06.701987 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:40:06.871364 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:06.871236 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5cf23bb-7230-4fa3-a127-195cec5093b3" path="/var/lib/kubelet/pods/a5cf23bb-7230-4fa3-a127-195cec5093b3/volumes" Sep 18 17:40:07.377203 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:07.377166 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:07.562251 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:07.562204 2560 patch_prober.go:28] interesting pod/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 18 17:40:07.562478 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:07.562283 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" podUID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 18 17:40:07.792487 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:07.792452 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:08.034592 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:08.034562 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:08.129406 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:08.129322 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:08.562515 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:08.562479 2560 patch_prober.go:28] interesting pod/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 18 17:40:08.563110 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:08.563078 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" podUID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 18 17:40:08.950629 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:08.950547 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:09.562180 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:09.562131 2560 patch_prober.go:28] interesting pod/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 18 17:40:09.562435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:09.562221 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" podUID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 18 17:40:09.641588 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:09.641540 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:10.561833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:10.561786 2560 patch_prober.go:28] interesting pod/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 18 17:40:10.562072 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:10.561876 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" podUID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 18 17:40:10.719600 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:10.719561 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:10.974833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:10.974793 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:11.075533 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:11.075494 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:11.210532 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:11.210501 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4"] Sep 18 17:40:11.210803 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:11.210772 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" podUID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerName="istio-proxy" containerID="cri-o://e84d06e8475019a7532167b02dd82141597decdafe98a5e644fa10260d6821e4" gracePeriod=30 Sep 18 17:40:11.232383 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:11.232268 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:11.730287 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:11.730250 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:11.736388 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:11.736353 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:40:11.737908 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:11.737878 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:40:13.031723 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:13.031691 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:13.731122 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:13.731083 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:14.597415 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:14.597380 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:15.259394 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:15.259362 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:16.036899 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.036866 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:16.473721 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.473696 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:40:16.597098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.596994 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-podinfo\") pod \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " Sep 18 17:40:16.597098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597059 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-envoy\") pod \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " Sep 18 17:40:16.597098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597096 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-data\") pod \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " Sep 18 17:40:16.597435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597155 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk674\" (UniqueName: \"kubernetes.io/projected/aac5040f-d169-4dcb-8a78-4b4e243b87a9-kube-api-access-pk674\") pod \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " Sep 18 17:40:16.597435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597200 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-workload-certs\") pod \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " Sep 18 17:40:16.597435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597214 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-token\") pod \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " Sep 18 17:40:16.597435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597273 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-workload-socket\") pod \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " Sep 18 17:40:16.597435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597343 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istiod-ca-cert\") pod \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " Sep 18 17:40:16.597435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597381 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-credential-socket\") pod \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\" (UID: \"aac5040f-d169-4dcb-8a78-4b4e243b87a9\") " Sep 18 17:40:16.597745 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597447 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-data" (OuterVolumeSpecName: "istio-data") pod "aac5040f-d169-4dcb-8a78-4b4e243b87a9" (UID: "aac5040f-d169-4dcb-8a78-4b4e243b87a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:16.597745 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597684 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aac5040f-d169-4dcb-8a78-4b4e243b87a9" (UID: "aac5040f-d169-4dcb-8a78-4b4e243b87a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:16.597860 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597780 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aac5040f-d169-4dcb-8a78-4b4e243b87a9" (UID: "aac5040f-d169-4dcb-8a78-4b4e243b87a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:16.597921 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597866 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:16.597921 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597885 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:16.597921 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.597899 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:16.598071 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.598044 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aac5040f-d169-4dcb-8a78-4b4e243b87a9" (UID: "aac5040f-d169-4dcb-8a78-4b4e243b87a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:16.598129 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.598067 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aac5040f-d169-4dcb-8a78-4b4e243b87a9" (UID: "aac5040f-d169-4dcb-8a78-4b4e243b87a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:40:16.599559 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.599531 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aac5040f-d169-4dcb-8a78-4b4e243b87a9" (UID: "aac5040f-d169-4dcb-8a78-4b4e243b87a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:40:16.599669 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.599549 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac5040f-d169-4dcb-8a78-4b4e243b87a9-kube-api-access-pk674" (OuterVolumeSpecName: "kube-api-access-pk674") pod "aac5040f-d169-4dcb-8a78-4b4e243b87a9" (UID: "aac5040f-d169-4dcb-8a78-4b4e243b87a9"). InnerVolumeSpecName "kube-api-access-pk674". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:40:16.599669 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.599613 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-token" (OuterVolumeSpecName: "istio-token") pod "aac5040f-d169-4dcb-8a78-4b4e243b87a9" (UID: "aac5040f-d169-4dcb-8a78-4b4e243b87a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:40:16.599669 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.599626 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aac5040f-d169-4dcb-8a78-4b4e243b87a9" (UID: "aac5040f-d169-4dcb-8a78-4b4e243b87a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:16.699147 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.699095 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:16.699147 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.699128 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:16.699147 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.699141 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk674\" (UniqueName: \"kubernetes.io/projected/aac5040f-d169-4dcb-8a78-4b4e243b87a9-kube-api-access-pk674\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:16.699147 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.699153 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:16.699462 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.699165 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac5040f-d169-4dcb-8a78-4b4e243b87a9-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:16.699462 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.699175 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac5040f-d169-4dcb-8a78-4b4e243b87a9-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:16.709566 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:16.709531 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:17.110791 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:17.110754 2560 generic.go:356] "Generic (PLEG): container finished" podID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerID="e84d06e8475019a7532167b02dd82141597decdafe98a5e644fa10260d6821e4" exitCode=0 Sep 18 17:40:17.110791 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:17.110798 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" Sep 18 17:40:17.111277 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:17.110832 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" event={"ID":"aac5040f-d169-4dcb-8a78-4b4e243b87a9","Type":"ContainerDied","Data":"e84d06e8475019a7532167b02dd82141597decdafe98a5e644fa10260d6821e4"} Sep 18 17:40:17.111277 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:17.110891 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4" event={"ID":"aac5040f-d169-4dcb-8a78-4b4e243b87a9","Type":"ContainerDied","Data":"6724899ba7e5491802d9cc0bb1097bd4b1c1b97e91ed51c096551038da2e988c"} Sep 18 17:40:17.111277 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:17.110918 2560 scope.go:117] "RemoveContainer" containerID="e84d06e8475019a7532167b02dd82141597decdafe98a5e644fa10260d6821e4" Sep 18 17:40:17.121161 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:17.121139 2560 scope.go:117] "RemoveContainer" containerID="07b9318e84864063432af940645ad9a4230220f7f0849c434860a75a900950d6" Sep 18 17:40:17.130506 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:17.130449 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4"] Sep 18 17:40:17.130694 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:17.130680 2560 scope.go:117] "RemoveContainer" containerID="e84d06e8475019a7532167b02dd82141597decdafe98a5e644fa10260d6821e4" Sep 18 17:40:17.131014 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:40:17.130992 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e84d06e8475019a7532167b02dd82141597decdafe98a5e644fa10260d6821e4\": container with ID starting with e84d06e8475019a7532167b02dd82141597decdafe98a5e644fa10260d6821e4 not found: ID does not exist" containerID="e84d06e8475019a7532167b02dd82141597decdafe98a5e644fa10260d6821e4" Sep 18 17:40:17.131088 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:17.131021 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e84d06e8475019a7532167b02dd82141597decdafe98a5e644fa10260d6821e4"} err="failed to get container status \"e84d06e8475019a7532167b02dd82141597decdafe98a5e644fa10260d6821e4\": rpc error: code = NotFound desc = could not find container \"e84d06e8475019a7532167b02dd82141597decdafe98a5e644fa10260d6821e4\": container with ID starting with e84d06e8475019a7532167b02dd82141597decdafe98a5e644fa10260d6821e4 not found: ID does not exist" Sep 18 17:40:17.131088 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:17.131038 2560 scope.go:117] "RemoveContainer" containerID="07b9318e84864063432af940645ad9a4230220f7f0849c434860a75a900950d6" Sep 18 17:40:17.131340 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:40:17.131313 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b9318e84864063432af940645ad9a4230220f7f0849c434860a75a900950d6\": container with ID starting with 07b9318e84864063432af940645ad9a4230220f7f0849c434860a75a900950d6 not found: ID does not exist" containerID="07b9318e84864063432af940645ad9a4230220f7f0849c434860a75a900950d6" Sep 18 17:40:17.131452 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:17.131348 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b9318e84864063432af940645ad9a4230220f7f0849c434860a75a900950d6"} err="failed to get container status \"07b9318e84864063432af940645ad9a4230220f7f0849c434860a75a900950d6\": rpc error: code = NotFound desc = could not find container \"07b9318e84864063432af940645ad9a4230220f7f0849c434860a75a900950d6\": container with ID starting with 07b9318e84864063432af940645ad9a4230220f7f0849c434860a75a900950d6 not found: ID does not exist" Sep 18 17:40:17.134731 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:17.134707 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9c0254-openshift-default-7647c8bb66-6nvx4"] Sep 18 17:40:17.394485 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:17.394364 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:18.537126 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:18.537047 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:18.866034 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:18.865952 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" path="/var/lib/kubelet/pods/aac5040f-d169-4dcb-8a78-4b4e243b87a9/volumes" Sep 18 17:40:21.253336 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:21.252584 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:21.350317 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:21.350262 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:21.910593 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:21.910556 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:40:21.912744 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:21.912720 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:27.327861 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.327818 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf"] Sep 18 17:40:27.328351 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.328081 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" podUID="2283838e-e81b-400a-a329-fbbd9b9dc75d" containerName="maas-api" containerID="cri-o://8f7b34d691b91dada39d0761eba5267c80dd5bcb1b273aa930cb62c67f1c8797" gracePeriod=30 Sep 18 17:40:27.359825 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.359787 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:27.454227 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.454187 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:27.538268 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.538237 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:27.598853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.598831 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:40:27.599001 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.598952 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:27.650684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.645650 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2283838e-e81b-400a-a329-fbbd9b9dc75d-metrics-certs\") pod \"2283838e-e81b-400a-a329-fbbd9b9dc75d\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " Sep 18 17:40:27.650684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.645718 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2283838e-e81b-400a-a329-fbbd9b9dc75d-maas-api-tls\") pod \"2283838e-e81b-400a-a329-fbbd9b9dc75d\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " Sep 18 17:40:27.650684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.645797 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txfsn\" (UniqueName: \"kubernetes.io/projected/2283838e-e81b-400a-a329-fbbd9b9dc75d-kube-api-access-txfsn\") pod \"2283838e-e81b-400a-a329-fbbd9b9dc75d\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " Sep 18 17:40:27.650684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.645867 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2283838e-e81b-400a-a329-fbbd9b9dc75d-openshift-service-ca\") pod \"2283838e-e81b-400a-a329-fbbd9b9dc75d\" (UID: \"2283838e-e81b-400a-a329-fbbd9b9dc75d\") " Sep 18 17:40:27.652324 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.652252 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2283838e-e81b-400a-a329-fbbd9b9dc75d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2283838e-e81b-400a-a329-fbbd9b9dc75d" (UID: "2283838e-e81b-400a-a329-fbbd9b9dc75d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:40:27.662745 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.662690 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2283838e-e81b-400a-a329-fbbd9b9dc75d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2283838e-e81b-400a-a329-fbbd9b9dc75d" (UID: "2283838e-e81b-400a-a329-fbbd9b9dc75d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:40:27.689596 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.689555 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2283838e-e81b-400a-a329-fbbd9b9dc75d-kube-api-access-txfsn" (OuterVolumeSpecName: "kube-api-access-txfsn") pod "2283838e-e81b-400a-a329-fbbd9b9dc75d" (UID: "2283838e-e81b-400a-a329-fbbd9b9dc75d"). InnerVolumeSpecName "kube-api-access-txfsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:40:27.692122 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.692076 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2283838e-e81b-400a-a329-fbbd9b9dc75d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2283838e-e81b-400a-a329-fbbd9b9dc75d" (UID: "2283838e-e81b-400a-a329-fbbd9b9dc75d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:40:27.746881 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.746846 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2283838e-e81b-400a-a329-fbbd9b9dc75d-metrics-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:27.746881 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.746875 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2283838e-e81b-400a-a329-fbbd9b9dc75d-maas-api-tls\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:27.746881 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.746885 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txfsn\" (UniqueName: \"kubernetes.io/projected/2283838e-e81b-400a-a329-fbbd9b9dc75d-kube-api-access-txfsn\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:27.747152 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.746894 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2283838e-e81b-400a-a329-fbbd9b9dc75d-openshift-service-ca\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:27.950225 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:27.950180 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:28.050889 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:28.050832 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:28.151421 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:28.151384 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:28.192962 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:28.192931 2560 generic.go:356] "Generic (PLEG): container finished" podID="2283838e-e81b-400a-a329-fbbd9b9dc75d" containerID="8f7b34d691b91dada39d0761eba5267c80dd5bcb1b273aa930cb62c67f1c8797" exitCode=0 Sep 18 17:40:28.193155 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:28.192976 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" Sep 18 17:40:28.193155 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:28.193010 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" event={"ID":"2283838e-e81b-400a-a329-fbbd9b9dc75d","Type":"ContainerDied","Data":"8f7b34d691b91dada39d0761eba5267c80dd5bcb1b273aa930cb62c67f1c8797"} Sep 18 17:40:28.193155 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:28.193060 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf" event={"ID":"2283838e-e81b-400a-a329-fbbd9b9dc75d","Type":"ContainerDied","Data":"e63e06d0db1bb626d21a13d6eadfe103d2cd9d4a5cf22226ae9f4066f7fce93b"} Sep 18 17:40:28.193155 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:28.193088 2560 scope.go:117] "RemoveContainer" containerID="8f7b34d691b91dada39d0761eba5267c80dd5bcb1b273aa930cb62c67f1c8797" Sep 18 17:40:28.203035 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:28.203016 2560 scope.go:117] "RemoveContainer" containerID="8f7b34d691b91dada39d0761eba5267c80dd5bcb1b273aa930cb62c67f1c8797" Sep 18 17:40:28.203358 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:40:28.203331 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7b34d691b91dada39d0761eba5267c80dd5bcb1b273aa930cb62c67f1c8797\": container with ID starting with 8f7b34d691b91dada39d0761eba5267c80dd5bcb1b273aa930cb62c67f1c8797 not found: ID does not exist" containerID="8f7b34d691b91dada39d0761eba5267c80dd5bcb1b273aa930cb62c67f1c8797" Sep 18 17:40:28.203423 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:28.203371 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7b34d691b91dada39d0761eba5267c80dd5bcb1b273aa930cb62c67f1c8797"} err="failed to get container status \"8f7b34d691b91dada39d0761eba5267c80dd5bcb1b273aa930cb62c67f1c8797\": rpc error: code = NotFound desc = could not find container \"8f7b34d691b91dada39d0761eba5267c80dd5bcb1b273aa930cb62c67f1c8797\": container with ID starting with 8f7b34d691b91dada39d0761eba5267c80dd5bcb1b273aa930cb62c67f1c8797 not found: ID does not exist" Sep 18 17:40:28.215853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:28.215823 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf"] Sep 18 17:40:28.219238 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:28.219212 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f4a54f-854ccc986c-kx9lf"] Sep 18 17:40:28.697424 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:28.697393 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:28.865566 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:28.865538 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2283838e-e81b-400a-a329-fbbd9b9dc75d" path="/var/lib/kubelet/pods/2283838e-e81b-400a-a329-fbbd9b9dc75d/volumes" Sep 18 17:40:29.443602 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:29.443560 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:30.127047 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:30.127016 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:30.870256 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:30.870220 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:31.624310 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:31.624263 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:34.328945 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:34.328906 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:34.431065 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:34.431034 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:34.575068 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:34.575031 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22"] Sep 18 17:40:34.575357 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:34.575331 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" podUID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerName="istio-proxy" containerID="cri-o://376e73792e309f8d19fa640e15b13e3d4e21939a8d2c5eb1aeff06c6d6aab24d" gracePeriod=30 Sep 18 17:40:34.591670 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:34.591589 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:35.103993 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:35.103961 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:36.132097 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:36.132065 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:36.838544 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:36.838514 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:37.517403 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:37.517371 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:38.218105 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:38.218071 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:38.897674 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:38.897587 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:39.634420 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.634390 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:39.850749 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.850667 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:40:39.986490 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.986453 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-data\") pod \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " Sep 18 17:40:39.986490 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.986486 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-envoy\") pod \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " Sep 18 17:40:39.987010 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.986525 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-podinfo\") pod \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " Sep 18 17:40:39.987010 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.986573 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istiod-ca-cert\") pod \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " Sep 18 17:40:39.987010 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.986595 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-token\") pod \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " Sep 18 17:40:39.987010 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.986647 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-credential-socket\") pod \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " Sep 18 17:40:39.987010 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.986677 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-workload-socket\") pod \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " Sep 18 17:40:39.987010 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.986707 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5qh9\" (UniqueName: \"kubernetes.io/projected/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-kube-api-access-f5qh9\") pod \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " Sep 18 17:40:39.987010 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.986768 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-workload-certs\") pod \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\" (UID: \"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068\") " Sep 18 17:40:39.987010 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.986776 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-data" (OuterVolumeSpecName: "istio-data") pod "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" (UID: "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:39.987010 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.986934 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" (UID: "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:39.987507 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.987021 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" (UID: "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:40:39.987507 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.987168 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" (UID: "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:39.987507 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.987340 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" (UID: "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:39.987507 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.987390 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:39.987507 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.987413 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:39.987507 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.987426 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:39.987507 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.987439 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:39.988810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.988790 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-token" (OuterVolumeSpecName: "istio-token") pod "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" (UID: "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:40:39.988961 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.988947 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-kube-api-access-f5qh9" (OuterVolumeSpecName: "kube-api-access-f5qh9") pod "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" (UID: "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068"). InnerVolumeSpecName "kube-api-access-f5qh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:40:39.989198 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.989181 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" (UID: "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:40:39.989249 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:39.989198 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" (UID: "d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:40.088394 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.088356 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:40.088394 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.088386 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:40.088394 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.088398 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:40.088641 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.088408 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:40.088641 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.088421 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5qh9\" (UniqueName: \"kubernetes.io/projected/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068-kube-api-access-f5qh9\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:40.270377 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.270341 2560 generic.go:356] "Generic (PLEG): container finished" podID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerID="376e73792e309f8d19fa640e15b13e3d4e21939a8d2c5eb1aeff06c6d6aab24d" exitCode=0 Sep 18 17:40:40.270575 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.270409 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" Sep 18 17:40:40.270575 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.270435 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" event={"ID":"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068","Type":"ContainerDied","Data":"376e73792e309f8d19fa640e15b13e3d4e21939a8d2c5eb1aeff06c6d6aab24d"} Sep 18 17:40:40.270575 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.270491 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22" event={"ID":"d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068","Type":"ContainerDied","Data":"aee57f611163b651f8aac66ee6381008fab758a14c9ed74215d48349af6366b1"} Sep 18 17:40:40.270575 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.270517 2560 scope.go:117] "RemoveContainer" containerID="376e73792e309f8d19fa640e15b13e3d4e21939a8d2c5eb1aeff06c6d6aab24d" Sep 18 17:40:40.282875 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.282857 2560 scope.go:117] "RemoveContainer" containerID="92dea9a0271c67642fb62ff5d82dd952ffa013072b5f931792be77f93644128b" Sep 18 17:40:40.291680 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.291663 2560 scope.go:117] "RemoveContainer" containerID="376e73792e309f8d19fa640e15b13e3d4e21939a8d2c5eb1aeff06c6d6aab24d" Sep 18 17:40:40.291954 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:40:40.291933 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"376e73792e309f8d19fa640e15b13e3d4e21939a8d2c5eb1aeff06c6d6aab24d\": container with ID starting with 376e73792e309f8d19fa640e15b13e3d4e21939a8d2c5eb1aeff06c6d6aab24d not found: ID does not exist" containerID="376e73792e309f8d19fa640e15b13e3d4e21939a8d2c5eb1aeff06c6d6aab24d" Sep 18 17:40:40.291990 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.291963 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"376e73792e309f8d19fa640e15b13e3d4e21939a8d2c5eb1aeff06c6d6aab24d"} err="failed to get container status \"376e73792e309f8d19fa640e15b13e3d4e21939a8d2c5eb1aeff06c6d6aab24d\": rpc error: code = NotFound desc = could not find container \"376e73792e309f8d19fa640e15b13e3d4e21939a8d2c5eb1aeff06c6d6aab24d\": container with ID starting with 376e73792e309f8d19fa640e15b13e3d4e21939a8d2c5eb1aeff06c6d6aab24d not found: ID does not exist" Sep 18 17:40:40.291990 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.291981 2560 scope.go:117] "RemoveContainer" containerID="92dea9a0271c67642fb62ff5d82dd952ffa013072b5f931792be77f93644128b" Sep 18 17:40:40.292205 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:40:40.292190 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92dea9a0271c67642fb62ff5d82dd952ffa013072b5f931792be77f93644128b\": container with ID starting with 92dea9a0271c67642fb62ff5d82dd952ffa013072b5f931792be77f93644128b not found: ID does not exist" containerID="92dea9a0271c67642fb62ff5d82dd952ffa013072b5f931792be77f93644128b" Sep 18 17:40:40.292238 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.292211 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92dea9a0271c67642fb62ff5d82dd952ffa013072b5f931792be77f93644128b"} err="failed to get container status \"92dea9a0271c67642fb62ff5d82dd952ffa013072b5f931792be77f93644128b\": rpc error: code = NotFound desc = could not find container \"92dea9a0271c67642fb62ff5d82dd952ffa013072b5f931792be77f93644128b\": container with ID starting with 92dea9a0271c67642fb62ff5d82dd952ffa013072b5f931792be77f93644128b not found: ID does not exist" Sep 18 17:40:40.297932 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.297704 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22"] Sep 18 17:40:40.300352 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.300332 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f4a54f-openshift-default-75c59f44b6-c9p22"] Sep 18 17:40:40.866089 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:40.866053 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" path="/var/lib/kubelet/pods/d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068/volumes" Sep 18 17:40:45.433461 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:45.433425 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:45.498752 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:45.498714 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:45.572824 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:45.572783 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:45.650922 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:45.650890 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:45.726719 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:45.726613 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:46.008802 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:46.008714 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:46.133179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:46.133148 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:46.966644 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:46.966608 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:48.097351 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:48.097316 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:49.041608 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:49.041558 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:49.722941 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:49.722905 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:50.713626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:50.713589 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:40:52.599551 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:52.599520 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:52.699920 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:52.699884 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:52.913788 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:52.913754 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn"] Sep 18 17:40:52.914015 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:52.913976 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" containerID="cri-o://99ee36409003a80576f0c5de1663ee42dd62e058d9a59f3baedcbfb290f23a4e" gracePeriod=30 Sep 18 17:40:52.918578 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:52.918555 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:53.538827 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:53.538787 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 18 17:40:53.539006 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:53.538850 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 18 17:40:53.559104 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:53.559074 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:54.596718 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:54.596685 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:55.381211 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:55.381180 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:56.136585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:56.136551 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:56.816194 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:56.816158 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:57.460696 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:57.460666 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:40:58.181346 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.181252 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:40:58.283419 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.283379 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istiod-ca-cert\") pod \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " Sep 18 17:40:58.283633 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.283438 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-credential-socket\") pod \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " Sep 18 17:40:58.283633 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.283508 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-token\") pod \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " Sep 18 17:40:58.283633 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.283565 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-data\") pod \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " Sep 18 17:40:58.283633 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.283593 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-podinfo\") pod \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " Sep 18 17:40:58.283633 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.283626 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-envoy\") pod \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " Sep 18 17:40:58.283894 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.283655 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5q59\" (UniqueName: \"kubernetes.io/projected/88400c22-6a60-4332-8a6e-c2d8d97eb76e-kube-api-access-q5q59\") pod \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " Sep 18 17:40:58.283894 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.283725 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-workload-certs\") pod \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " Sep 18 17:40:58.283894 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.283763 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-workload-socket\") pod \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\" (UID: \"88400c22-6a60-4332-8a6e-c2d8d97eb76e\") " Sep 18 17:40:58.283894 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.283789 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88400c22-6a60-4332-8a6e-c2d8d97eb76e" (UID: "88400c22-6a60-4332-8a6e-c2d8d97eb76e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:58.284105 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.283897 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-data" (OuterVolumeSpecName: "istio-data") pod "88400c22-6a60-4332-8a6e-c2d8d97eb76e" (UID: "88400c22-6a60-4332-8a6e-c2d8d97eb76e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:58.284105 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.283825 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88400c22-6a60-4332-8a6e-c2d8d97eb76e" (UID: "88400c22-6a60-4332-8a6e-c2d8d97eb76e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:40:58.284105 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.284075 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88400c22-6a60-4332-8a6e-c2d8d97eb76e" (UID: "88400c22-6a60-4332-8a6e-c2d8d97eb76e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:58.284258 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.284112 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88400c22-6a60-4332-8a6e-c2d8d97eb76e" (UID: "88400c22-6a60-4332-8a6e-c2d8d97eb76e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:58.284360 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.284273 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:58.284360 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.284310 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:58.284360 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.284323 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:58.284360 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.284336 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:58.284360 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.284349 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:58.285923 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.285902 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-token" (OuterVolumeSpecName: "istio-token") pod "88400c22-6a60-4332-8a6e-c2d8d97eb76e" (UID: "88400c22-6a60-4332-8a6e-c2d8d97eb76e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:40:58.286008 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.285903 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88400c22-6a60-4332-8a6e-c2d8d97eb76e" (UID: "88400c22-6a60-4332-8a6e-c2d8d97eb76e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:40:58.286206 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.286185 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88400c22-6a60-4332-8a6e-c2d8d97eb76e" (UID: "88400c22-6a60-4332-8a6e-c2d8d97eb76e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:40:58.286328 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.286218 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88400c22-6a60-4332-8a6e-c2d8d97eb76e-kube-api-access-q5q59" (OuterVolumeSpecName: "kube-api-access-q5q59") pod "88400c22-6a60-4332-8a6e-c2d8d97eb76e" (UID: "88400c22-6a60-4332-8a6e-c2d8d97eb76e"). InnerVolumeSpecName "kube-api-access-q5q59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:40:58.386135 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.385758 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:58.386135 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.385805 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:58.386135 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.385818 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5q59\" (UniqueName: \"kubernetes.io/projected/88400c22-6a60-4332-8a6e-c2d8d97eb76e-kube-api-access-q5q59\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:58.386135 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.385832 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88400c22-6a60-4332-8a6e-c2d8d97eb76e-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:40:58.388321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.388260 2560 generic.go:356] "Generic (PLEG): container finished" podID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerID="99ee36409003a80576f0c5de1663ee42dd62e058d9a59f3baedcbfb290f23a4e" exitCode=0 Sep 18 17:40:58.388460 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.388331 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" event={"ID":"88400c22-6a60-4332-8a6e-c2d8d97eb76e","Type":"ContainerDied","Data":"99ee36409003a80576f0c5de1663ee42dd62e058d9a59f3baedcbfb290f23a4e"} Sep 18 17:40:58.388460 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.388368 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" Sep 18 17:40:58.388460 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.388414 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn" event={"ID":"88400c22-6a60-4332-8a6e-c2d8d97eb76e","Type":"ContainerDied","Data":"64ea62d5367369f56ca3cc206b806effa8fd2f76fe1fbf30168d4d18102658bf"} Sep 18 17:40:58.388460 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.388439 2560 scope.go:117] "RemoveContainer" containerID="99ee36409003a80576f0c5de1663ee42dd62e058d9a59f3baedcbfb290f23a4e" Sep 18 17:40:58.401581 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.401552 2560 scope.go:117] "RemoveContainer" containerID="0286f57963d9e07f55799cb22d4950a5a4231ebf17ffdb0e5dae3bc7cb266e00" Sep 18 17:40:58.411640 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.411614 2560 scope.go:117] "RemoveContainer" containerID="99ee36409003a80576f0c5de1663ee42dd62e058d9a59f3baedcbfb290f23a4e" Sep 18 17:40:58.411947 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:40:58.411927 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ee36409003a80576f0c5de1663ee42dd62e058d9a59f3baedcbfb290f23a4e\": container with ID starting with 99ee36409003a80576f0c5de1663ee42dd62e058d9a59f3baedcbfb290f23a4e not found: ID does not exist" containerID="99ee36409003a80576f0c5de1663ee42dd62e058d9a59f3baedcbfb290f23a4e" Sep 18 17:40:58.412003 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.411957 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ee36409003a80576f0c5de1663ee42dd62e058d9a59f3baedcbfb290f23a4e"} err="failed to get container status \"99ee36409003a80576f0c5de1663ee42dd62e058d9a59f3baedcbfb290f23a4e\": rpc error: code = NotFound desc = could not find container \"99ee36409003a80576f0c5de1663ee42dd62e058d9a59f3baedcbfb290f23a4e\": container with ID starting with 99ee36409003a80576f0c5de1663ee42dd62e058d9a59f3baedcbfb290f23a4e not found: ID does not exist" Sep 18 17:40:58.412003 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.411977 2560 scope.go:117] "RemoveContainer" containerID="0286f57963d9e07f55799cb22d4950a5a4231ebf17ffdb0e5dae3bc7cb266e00" Sep 18 17:40:58.412260 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:40:58.412238 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0286f57963d9e07f55799cb22d4950a5a4231ebf17ffdb0e5dae3bc7cb266e00\": container with ID starting with 0286f57963d9e07f55799cb22d4950a5a4231ebf17ffdb0e5dae3bc7cb266e00 not found: ID does not exist" containerID="0286f57963d9e07f55799cb22d4950a5a4231ebf17ffdb0e5dae3bc7cb266e00" Sep 18 17:40:58.412330 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.412267 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0286f57963d9e07f55799cb22d4950a5a4231ebf17ffdb0e5dae3bc7cb266e00"} err="failed to get container status \"0286f57963d9e07f55799cb22d4950a5a4231ebf17ffdb0e5dae3bc7cb266e00\": rpc error: code = NotFound desc = could not find container \"0286f57963d9e07f55799cb22d4950a5a4231ebf17ffdb0e5dae3bc7cb266e00\": container with ID starting with 0286f57963d9e07f55799cb22d4950a5a4231ebf17ffdb0e5dae3bc7cb266e00 not found: ID does not exist" Sep 18 17:40:58.424214 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.424178 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn"] Sep 18 17:40:58.429725 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.429694 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-dd235c-openshift-default-6657ffb5dd-k2ssn"] Sep 18 17:40:58.865673 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:40:58.865630 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" path="/var/lib/kubelet/pods/88400c22-6a60-4332-8a6e-c2d8d97eb76e/volumes" Sep 18 17:41:03.769032 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:03.768990 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:03.880164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:03.880135 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:03.981740 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:03.981694 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:04.090237 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:04.090153 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:04.182968 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:04.182930 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:04.455149 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:04.455114 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:04.563479 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:04.563449 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:05.767982 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:05.767949 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:06.485472 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:06.485436 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:07.591585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:07.591546 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:11.163986 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:11.163952 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:11.226970 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:11.226937 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:11.417407 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:11.417272 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc"] Sep 18 17:41:11.417852 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:11.417823 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" podUID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerName="istio-proxy" containerID="cri-o://0b1928dafa2c04115531bf8dff8fbdf0d8772de313fc10b7df22a59abef6761a" gracePeriod=30 Sep 18 17:41:11.439129 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:11.439099 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:16.683204 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.683180 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:41:16.791615 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.791530 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-workload-socket\") pod \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " Sep 18 17:41:16.791615 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.791565 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-workload-certs\") pod \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " Sep 18 17:41:16.791615 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.791608 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-credential-socket\") pod \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " Sep 18 17:41:16.791883 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.791654 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istiod-ca-cert\") pod \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " Sep 18 17:41:16.791883 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.791680 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-token\") pod \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " Sep 18 17:41:16.791883 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.791694 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd9ck\" (UniqueName: \"kubernetes.io/projected/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-kube-api-access-cd9ck\") pod \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " Sep 18 17:41:16.791883 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.791731 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-data\") pod \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " Sep 18 17:41:16.792111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.791886 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-envoy\") pod \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " Sep 18 17:41:16.792111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.791884 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" (UID: "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:16.792111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.791951 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-podinfo\") pod \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\" (UID: \"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183\") " Sep 18 17:41:16.792111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.791897 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" (UID: "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:16.792111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.791940 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" (UID: "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:16.792111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.792030 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" (UID: "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:16.792469 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.792313 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" (UID: "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:41:16.792532 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.792476 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:16.792532 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.792495 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:16.792532 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.792504 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:16.792532 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.792513 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:16.792532 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.792521 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:16.793972 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.793936 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-kube-api-access-cd9ck" (OuterVolumeSpecName: "kube-api-access-cd9ck") pod "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" (UID: "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183"). InnerVolumeSpecName "kube-api-access-cd9ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:41:16.794085 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.794050 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" (UID: "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:41:16.794359 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.794342 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" (UID: "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:41:16.794435 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.794357 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" (UID: "6f6408dd-e3ae-4a25-b34c-af0a8d3b0183"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:16.893477 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.893450 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:16.893477 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.893475 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:16.893477 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.893483 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd9ck\" (UniqueName: \"kubernetes.io/projected/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-kube-api-access-cd9ck\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:16.893695 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:16.893493 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:17.514564 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:17.514527 2560 generic.go:356] "Generic (PLEG): container finished" podID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerID="0b1928dafa2c04115531bf8dff8fbdf0d8772de313fc10b7df22a59abef6761a" exitCode=0 Sep 18 17:41:17.514739 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:17.514572 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" Sep 18 17:41:17.514739 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:17.514610 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" event={"ID":"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183","Type":"ContainerDied","Data":"0b1928dafa2c04115531bf8dff8fbdf0d8772de313fc10b7df22a59abef6761a"} Sep 18 17:41:17.514739 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:17.514669 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc" event={"ID":"6f6408dd-e3ae-4a25-b34c-af0a8d3b0183","Type":"ContainerDied","Data":"ee9009367e36f26916815ff1e2b6fb6d4bb891f890873a5c3c67aa6dd4563abb"} Sep 18 17:41:17.514739 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:17.514693 2560 scope.go:117] "RemoveContainer" containerID="0b1928dafa2c04115531bf8dff8fbdf0d8772de313fc10b7df22a59abef6761a" Sep 18 17:41:17.524445 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:17.524427 2560 scope.go:117] "RemoveContainer" containerID="52259bd42f462191dd29014f4df106e157a69da708316323d089558dac0459b8" Sep 18 17:41:17.533549 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:17.533525 2560 scope.go:117] "RemoveContainer" containerID="0b1928dafa2c04115531bf8dff8fbdf0d8772de313fc10b7df22a59abef6761a" Sep 18 17:41:17.533943 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:41:17.533920 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1928dafa2c04115531bf8dff8fbdf0d8772de313fc10b7df22a59abef6761a\": container with ID starting with 0b1928dafa2c04115531bf8dff8fbdf0d8772de313fc10b7df22a59abef6761a not found: ID does not exist" containerID="0b1928dafa2c04115531bf8dff8fbdf0d8772de313fc10b7df22a59abef6761a" Sep 18 17:41:17.534040 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:17.533950 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1928dafa2c04115531bf8dff8fbdf0d8772de313fc10b7df22a59abef6761a"} err="failed to get container status \"0b1928dafa2c04115531bf8dff8fbdf0d8772de313fc10b7df22a59abef6761a\": rpc error: code = NotFound desc = could not find container \"0b1928dafa2c04115531bf8dff8fbdf0d8772de313fc10b7df22a59abef6761a\": container with ID starting with 0b1928dafa2c04115531bf8dff8fbdf0d8772de313fc10b7df22a59abef6761a not found: ID does not exist" Sep 18 17:41:17.534040 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:17.533969 2560 scope.go:117] "RemoveContainer" containerID="52259bd42f462191dd29014f4df106e157a69da708316323d089558dac0459b8" Sep 18 17:41:17.534277 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:41:17.534259 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52259bd42f462191dd29014f4df106e157a69da708316323d089558dac0459b8\": container with ID starting with 52259bd42f462191dd29014f4df106e157a69da708316323d089558dac0459b8 not found: ID does not exist" containerID="52259bd42f462191dd29014f4df106e157a69da708316323d089558dac0459b8" Sep 18 17:41:17.534351 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:17.534282 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52259bd42f462191dd29014f4df106e157a69da708316323d089558dac0459b8"} err="failed to get container status \"52259bd42f462191dd29014f4df106e157a69da708316323d089558dac0459b8\": rpc error: code = NotFound desc = could not find container \"52259bd42f462191dd29014f4df106e157a69da708316323d089558dac0459b8\": container with ID starting with 52259bd42f462191dd29014f4df106e157a69da708316323d089558dac0459b8 not found: ID does not exist" Sep 18 17:41:17.534981 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:17.534955 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc"] Sep 18 17:41:17.538044 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:17.538024 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-66efa6-openshift-default-6b68cf4696-nkxwc"] Sep 18 17:41:18.865866 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:18.865832 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" path="/var/lib/kubelet/pods/6f6408dd-e3ae-4a25-b34c-af0a8d3b0183/volumes" Sep 18 17:41:19.025885 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:19.025848 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:41:24.963996 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:24.963963 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:26.151537 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:26.151492 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:27.023259 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:27.023186 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:27.644950 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:27.644908 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:28.337201 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:28.337172 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:29.949262 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:29.949178 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk"] Sep 18 17:41:30.042144 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:30.042114 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:40.921470 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:40.921436 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:41.012708 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:41.012675 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:41.097702 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:41.097673 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:41.157635 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:41.157603 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:41.312382 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:41.312286 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:41:41.468189 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:41.468157 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:41.594727 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:41.594638 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:42.740803 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:42.740772 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:43.557576 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:43.557542 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:44.970308 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:44.970258 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" podUID="22816025-feac-448a-a289-063138c3d64c" containerName="main" containerID="cri-o://3d3d47967f49d266620f660c32b203cf69c5f49229fdfc7ee7df5b2e67954144" gracePeriod=45 Sep 18 17:41:45.228984 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.228927 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:41:45.295040 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.295011 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22816025-feac-448a-a289-063138c3d64c-tls-certs\") pod \"22816025-feac-448a-a289-063138c3d64c\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " Sep 18 17:41:45.295224 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.295061 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pstlf\" (UniqueName: \"kubernetes.io/projected/22816025-feac-448a-a289-063138c3d64c-kube-api-access-pstlf\") pod \"22816025-feac-448a-a289-063138c3d64c\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " Sep 18 17:41:45.295224 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.295089 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-tmp-dir\") pod \"22816025-feac-448a-a289-063138c3d64c\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " Sep 18 17:41:45.295224 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.295118 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-dshm\") pod \"22816025-feac-448a-a289-063138c3d64c\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " Sep 18 17:41:45.295224 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.295150 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-home\") pod \"22816025-feac-448a-a289-063138c3d64c\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " Sep 18 17:41:45.295469 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.295305 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-model-cache\") pod \"22816025-feac-448a-a289-063138c3d64c\" (UID: \"22816025-feac-448a-a289-063138c3d64c\") " Sep 18 17:41:45.295469 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.295393 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22816025-feac-448a-a289-063138c3d64c" (UID: "22816025-feac-448a-a289-063138c3d64c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:45.295589 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.295455 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-home" (OuterVolumeSpecName: "home") pod "22816025-feac-448a-a289-063138c3d64c" (UID: "22816025-feac-448a-a289-063138c3d64c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:45.295703 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.295682 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-model-cache" (OuterVolumeSpecName: "model-cache") pod "22816025-feac-448a-a289-063138c3d64c" (UID: "22816025-feac-448a-a289-063138c3d64c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:45.295773 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.295700 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-tmp-dir\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:45.295773 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.295721 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-home\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:45.297494 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.297471 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-dshm" (OuterVolumeSpecName: "dshm") pod "22816025-feac-448a-a289-063138c3d64c" (UID: "22816025-feac-448a-a289-063138c3d64c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:45.297494 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.297484 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22816025-feac-448a-a289-063138c3d64c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22816025-feac-448a-a289-063138c3d64c" (UID: "22816025-feac-448a-a289-063138c3d64c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:41:45.297645 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.297595 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22816025-feac-448a-a289-063138c3d64c-kube-api-access-pstlf" (OuterVolumeSpecName: "kube-api-access-pstlf") pod "22816025-feac-448a-a289-063138c3d64c" (UID: "22816025-feac-448a-a289-063138c3d64c"). InnerVolumeSpecName "kube-api-access-pstlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:41:45.396374 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.396342 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pstlf\" (UniqueName: \"kubernetes.io/projected/22816025-feac-448a-a289-063138c3d64c-kube-api-access-pstlf\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:45.396374 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.396370 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-dshm\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:45.396374 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.396381 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22816025-feac-448a-a289-063138c3d64c-model-cache\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:45.396610 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.396389 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22816025-feac-448a-a289-063138c3d64c-tls-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:45.690382 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.690351 2560 generic.go:356] "Generic (PLEG): container finished" podID="22816025-feac-448a-a289-063138c3d64c" containerID="3d3d47967f49d266620f660c32b203cf69c5f49229fdfc7ee7df5b2e67954144" exitCode=0 Sep 18 17:41:45.690571 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.690391 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" Sep 18 17:41:45.690571 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.690395 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" event={"ID":"22816025-feac-448a-a289-063138c3d64c","Type":"ContainerDied","Data":"3d3d47967f49d266620f660c32b203cf69c5f49229fdfc7ee7df5b2e67954144"} Sep 18 17:41:45.690571 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.690421 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk" event={"ID":"22816025-feac-448a-a289-063138c3d64c","Type":"ContainerDied","Data":"78845c44f5ed780a02f632328d59a7a314cc206409c5fb7e157aca176a6f684f"} Sep 18 17:41:45.690571 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.690439 2560 scope.go:117] "RemoveContainer" containerID="3d3d47967f49d266620f660c32b203cf69c5f49229fdfc7ee7df5b2e67954144" Sep 18 17:41:45.700281 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.700257 2560 scope.go:117] "RemoveContainer" containerID="3d3d47967f49d266620f660c32b203cf69c5f49229fdfc7ee7df5b2e67954144" Sep 18 17:41:45.700608 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:41:45.700590 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d3d47967f49d266620f660c32b203cf69c5f49229fdfc7ee7df5b2e67954144\": container with ID starting with 3d3d47967f49d266620f660c32b203cf69c5f49229fdfc7ee7df5b2e67954144 not found: ID does not exist" containerID="3d3d47967f49d266620f660c32b203cf69c5f49229fdfc7ee7df5b2e67954144" Sep 18 17:41:45.700685 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.700618 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d3d47967f49d266620f660c32b203cf69c5f49229fdfc7ee7df5b2e67954144"} err="failed to get container status \"3d3d47967f49d266620f660c32b203cf69c5f49229fdfc7ee7df5b2e67954144\": rpc error: code = NotFound desc = could not find container \"3d3d47967f49d266620f660c32b203cf69c5f49229fdfc7ee7df5b2e67954144\": container with ID starting with 3d3d47967f49d266620f660c32b203cf69c5f49229fdfc7ee7df5b2e67954144 not found: ID does not exist" Sep 18 17:41:45.712286 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.712260 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk"] Sep 18 17:41:45.719184 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:45.719146 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-5b6d69/distinct-2-w4-5b6d69-kserve-74cdf64596-sznxk"] Sep 18 17:41:46.865770 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:46.865733 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22816025-feac-448a-a289-063138c3d64c" path="/var/lib/kubelet/pods/22816025-feac-448a-a289-063138c3d64c/volumes" Sep 18 17:41:47.011370 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:47.011330 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:48.015670 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:48.015643 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:48.401878 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:48.401851 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:48.563974 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:48.563938 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:48.761529 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:48.761440 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:48.773537 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:48.773497 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7"] Sep 18 17:41:48.773746 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:48.773717 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" containerID="cri-o://77599988ecb09e76cdb39982ba71f98bd489e408ea60af4250fc6ccf4e5c37bb" gracePeriod=30 Sep 18 17:41:52.949226 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:52.949164 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:41:54.027154 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.027131 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:41:54.190425 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.190386 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84m6b\" (UniqueName: \"kubernetes.io/projected/ad8c0339-0a9d-462f-922e-2359d1ead55b-kube-api-access-84m6b\") pod \"ad8c0339-0a9d-462f-922e-2359d1ead55b\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " Sep 18 17:41:54.190621 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.190447 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-podinfo\") pod \"ad8c0339-0a9d-462f-922e-2359d1ead55b\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " Sep 18 17:41:54.190621 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.190465 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8c0339-0a9d-462f-922e-2359d1ead55b-istiod-ca-cert\") pod \"ad8c0339-0a9d-462f-922e-2359d1ead55b\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " Sep 18 17:41:54.190621 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.190524 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-envoy\") pod \"ad8c0339-0a9d-462f-922e-2359d1ead55b\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " Sep 18 17:41:54.190621 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.190578 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-workload-socket\") pod \"ad8c0339-0a9d-462f-922e-2359d1ead55b\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " Sep 18 17:41:54.190621 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.190614 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-data\") pod \"ad8c0339-0a9d-462f-922e-2359d1ead55b\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " Sep 18 17:41:54.190897 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.190647 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-credential-socket\") pod \"ad8c0339-0a9d-462f-922e-2359d1ead55b\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " Sep 18 17:41:54.190897 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.190670 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-workload-certs\") pod \"ad8c0339-0a9d-462f-922e-2359d1ead55b\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " Sep 18 17:41:54.190897 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.190742 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-token\") pod \"ad8c0339-0a9d-462f-922e-2359d1ead55b\" (UID: \"ad8c0339-0a9d-462f-922e-2359d1ead55b\") " Sep 18 17:41:54.190897 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.190825 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ad8c0339-0a9d-462f-922e-2359d1ead55b" (UID: "ad8c0339-0a9d-462f-922e-2359d1ead55b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:54.191114 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.191013 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ad8c0339-0a9d-462f-922e-2359d1ead55b" (UID: "ad8c0339-0a9d-462f-922e-2359d1ead55b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:54.191114 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.191032 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-data" (OuterVolumeSpecName: "istio-data") pod "ad8c0339-0a9d-462f-922e-2359d1ead55b" (UID: "ad8c0339-0a9d-462f-922e-2359d1ead55b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:54.191225 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.191172 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ad8c0339-0a9d-462f-922e-2359d1ead55b" (UID: "ad8c0339-0a9d-462f-922e-2359d1ead55b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:54.191288 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.191274 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8c0339-0a9d-462f-922e-2359d1ead55b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ad8c0339-0a9d-462f-922e-2359d1ead55b" (UID: "ad8c0339-0a9d-462f-922e-2359d1ead55b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:41:54.191379 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.191309 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:54.191379 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.191330 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:54.191379 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.191343 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:54.192929 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.192899 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8c0339-0a9d-462f-922e-2359d1ead55b-kube-api-access-84m6b" (OuterVolumeSpecName: "kube-api-access-84m6b") pod "ad8c0339-0a9d-462f-922e-2359d1ead55b" (UID: "ad8c0339-0a9d-462f-922e-2359d1ead55b"). InnerVolumeSpecName "kube-api-access-84m6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:41:54.193056 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.193010 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ad8c0339-0a9d-462f-922e-2359d1ead55b" (UID: "ad8c0339-0a9d-462f-922e-2359d1ead55b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:41:54.193119 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.193099 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ad8c0339-0a9d-462f-922e-2359d1ead55b" (UID: "ad8c0339-0a9d-462f-922e-2359d1ead55b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:41:54.193159 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.193127 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad8c0339-0a9d-462f-922e-2359d1ead55b" (UID: "ad8c0339-0a9d-462f-922e-2359d1ead55b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:41:54.292545 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.292505 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:54.292545 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.292538 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8c0339-0a9d-462f-922e-2359d1ead55b-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:54.292545 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.292547 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:54.292545 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.292556 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84m6b\" (UniqueName: \"kubernetes.io/projected/ad8c0339-0a9d-462f-922e-2359d1ead55b-kube-api-access-84m6b\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:54.292825 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.292565 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8c0339-0a9d-462f-922e-2359d1ead55b-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:54.292825 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.292574 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8c0339-0a9d-462f-922e-2359d1ead55b-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:41:54.746396 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.746357 2560 generic.go:356] "Generic (PLEG): container finished" podID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerID="77599988ecb09e76cdb39982ba71f98bd489e408ea60af4250fc6ccf4e5c37bb" exitCode=0 Sep 18 17:41:54.746396 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.746403 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" Sep 18 17:41:54.746727 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.746449 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" event={"ID":"ad8c0339-0a9d-462f-922e-2359d1ead55b","Type":"ContainerDied","Data":"77599988ecb09e76cdb39982ba71f98bd489e408ea60af4250fc6ccf4e5c37bb"} Sep 18 17:41:54.746727 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.746510 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7" event={"ID":"ad8c0339-0a9d-462f-922e-2359d1ead55b","Type":"ContainerDied","Data":"3e78e64c4e8fe611ce079b7e7d2a91a1393b43e9c88fd053271281b16fc0eadd"} Sep 18 17:41:54.746727 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.746542 2560 scope.go:117] "RemoveContainer" containerID="77599988ecb09e76cdb39982ba71f98bd489e408ea60af4250fc6ccf4e5c37bb" Sep 18 17:41:54.758745 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.758724 2560 scope.go:117] "RemoveContainer" containerID="535261ffb1094ba1a880bb0348595c251a2ccd8a239c54cd32b162ed8ade2cce" Sep 18 17:41:54.767270 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.767252 2560 scope.go:117] "RemoveContainer" containerID="77599988ecb09e76cdb39982ba71f98bd489e408ea60af4250fc6ccf4e5c37bb" Sep 18 17:41:54.767565 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:41:54.767546 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77599988ecb09e76cdb39982ba71f98bd489e408ea60af4250fc6ccf4e5c37bb\": container with ID starting with 77599988ecb09e76cdb39982ba71f98bd489e408ea60af4250fc6ccf4e5c37bb not found: ID does not exist" containerID="77599988ecb09e76cdb39982ba71f98bd489e408ea60af4250fc6ccf4e5c37bb" Sep 18 17:41:54.767619 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.767574 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77599988ecb09e76cdb39982ba71f98bd489e408ea60af4250fc6ccf4e5c37bb"} err="failed to get container status \"77599988ecb09e76cdb39982ba71f98bd489e408ea60af4250fc6ccf4e5c37bb\": rpc error: code = NotFound desc = could not find container \"77599988ecb09e76cdb39982ba71f98bd489e408ea60af4250fc6ccf4e5c37bb\": container with ID starting with 77599988ecb09e76cdb39982ba71f98bd489e408ea60af4250fc6ccf4e5c37bb not found: ID does not exist" Sep 18 17:41:54.767619 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.767592 2560 scope.go:117] "RemoveContainer" containerID="535261ffb1094ba1a880bb0348595c251a2ccd8a239c54cd32b162ed8ade2cce" Sep 18 17:41:54.767830 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:41:54.767812 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535261ffb1094ba1a880bb0348595c251a2ccd8a239c54cd32b162ed8ade2cce\": container with ID starting with 535261ffb1094ba1a880bb0348595c251a2ccd8a239c54cd32b162ed8ade2cce not found: ID does not exist" containerID="535261ffb1094ba1a880bb0348595c251a2ccd8a239c54cd32b162ed8ade2cce" Sep 18 17:41:54.767868 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.767834 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535261ffb1094ba1a880bb0348595c251a2ccd8a239c54cd32b162ed8ade2cce"} err="failed to get container status \"535261ffb1094ba1a880bb0348595c251a2ccd8a239c54cd32b162ed8ade2cce\": rpc error: code = NotFound desc = could not find container \"535261ffb1094ba1a880bb0348595c251a2ccd8a239c54cd32b162ed8ade2cce\": container with ID starting with 535261ffb1094ba1a880bb0348595c251a2ccd8a239c54cd32b162ed8ade2cce not found: ID does not exist" Sep 18 17:41:54.772382 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.772359 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7"] Sep 18 17:41:54.774341 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.774317 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-5b6d69-openshift-default-678dbd8cdd-kxkj7"] Sep 18 17:41:54.865406 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:54.865373 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" path="/var/lib/kubelet/pods/ad8c0339-0a9d-462f-922e-2359d1ead55b/volumes" Sep 18 17:41:56.733770 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:41:56.733734 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:42:00.308253 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:42:00.308214 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:42:07.391218 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:42:07.391185 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:42:07.491201 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:42:07.491168 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:42:08.029896 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:42:08.029847 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:42:14.715530 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:42:14.715496 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:42:20.557340 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:42:20.557287 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:42:22.926253 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:42:22.926212 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:42:25.139225 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:42:25.139195 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:42:26.528877 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:42:26.528839 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:42:51.520179 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:42:51.520142 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:43:05.308751 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:43:05.308652 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:43:15.759954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:43:15.759915 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:43:16.731855 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:43:16.731809 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:43:19.019934 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:43:19.019893 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:43:28.361136 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:43:28.361053 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:43:31.419825 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:43:31.419788 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:43:47.816565 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:43:47.816492 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:43:51.606091 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:43:51.606057 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:43:56.491919 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:43:56.491889 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:30.252572 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:30.252472 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:35.343510 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:35.343473 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:44:44.835111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:44.835072 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:44:48.728396 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:48.728334 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:48.999410 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:48.999321 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:44:49.004524 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:49.004500 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:44:49.005737 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:49.005714 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:44:49.010566 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:49.010548 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:44:51.353739 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:51.353694 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:52.447286 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:52.447236 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:52.856011 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:52.855902 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:53.186756 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:53.186721 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:53.382374 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:53.382325 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:55.502888 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:55.502845 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:44:56.428225 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:56.428178 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:56.514407 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:56.514372 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:57.413373 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:57.413345 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:58.432437 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:58.432404 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:58.530475 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:58.530436 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:58.628427 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:58.628393 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:58.915537 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:58.915502 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:44:58.997018 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:44:58.996987 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:00.149421 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.149385 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs"] Sep 18 17:45:00.150078 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150057 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerName="istio-proxy" Sep 18 17:45:00.150078 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150081 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerName="istio-proxy" Sep 18 17:45:00.150231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150094 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5cf23bb-7230-4fa3-a127-195cec5093b3" containerName="maas-api" Sep 18 17:45:00.150231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150102 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5cf23bb-7230-4fa3-a127-195cec5093b3" containerName="maas-api" Sep 18 17:45:00.150231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150114 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerName="istio-proxy" Sep 18 17:45:00.150231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150124 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerName="istio-proxy" Sep 18 17:45:00.150231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150145 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" Sep 18 17:45:00.150231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150154 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" Sep 18 17:45:00.150231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150169 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22816025-feac-448a-a289-063138c3d64c" containerName="main" Sep 18 17:45:00.150231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150177 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="22816025-feac-448a-a289-063138c3d64c" containerName="main" Sep 18 17:45:00.150231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150194 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerName="istio-proxy" Sep 18 17:45:00.150231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150203 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerName="istio-proxy" Sep 18 17:45:00.150231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150212 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" Sep 18 17:45:00.150231 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150221 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150243 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150252 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150264 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e3b4dba-f46c-443d-9728-3a3fb1afe129" containerName="payload-processing" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150272 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e3b4dba-f46c-443d-9728-3a3fb1afe129" containerName="payload-processing" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150283 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150307 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150331 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2283838e-e81b-400a-a329-fbbd9b9dc75d" containerName="maas-api" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150340 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2283838e-e81b-400a-a329-fbbd9b9dc75d" containerName="maas-api" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150448 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150460 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150471 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150481 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="88400c22-6a60-4332-8a6e-c2d8d97eb76e" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150489 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e3b4dba-f46c-443d-9728-3a3fb1afe129" containerName="payload-processing" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150497 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5cf23bb-7230-4fa3-a127-195cec5093b3" containerName="maas-api" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150507 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150514 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150523 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150534 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150543 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8c0339-0a9d-462f-922e-2359d1ead55b" containerName="istio-proxy" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150553 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="22816025-feac-448a-a289-063138c3d64c" containerName="main" Sep 18 17:45:00.150858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.150563 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2283838e-e81b-400a-a329-fbbd9b9dc75d" containerName="maas-api" Sep 18 17:45:00.154337 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.154313 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs" Sep 18 17:45:00.157951 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.157930 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qt978\"" Sep 18 17:45:00.165267 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.165245 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs" Sep 18 17:45:00.184166 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.184131 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs"] Sep 18 17:45:00.448641 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.448608 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs"] Sep 18 17:45:00.450268 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:45:00.450242 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e116d6_e2f0_46c9_bbf6_053c0acafa6b.slice/crio-9b395d0c0c9f725970e1a91dca6dd58ac436512900b912e7591b3e351e470607 WatchSource:0}: Error finding container 9b395d0c0c9f725970e1a91dca6dd58ac436512900b912e7591b3e351e470607: Status 404 returned error can't find the container with id 9b395d0c0c9f725970e1a91dca6dd58ac436512900b912e7591b3e351e470607 Sep 18 17:45:00.809613 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.809573 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs" event={"ID":"74e116d6-e2f0-46c9-bbf6-053c0acafa6b","Type":"ContainerStarted","Data":"72c7e81d89141de727616a17d5c78370fb82d1534fc4e2d8cccf50d76781fa4e"} Sep 18 17:45:00.809791 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.809637 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs" event={"ID":"74e116d6-e2f0-46c9-bbf6-053c0acafa6b","Type":"ContainerStarted","Data":"9b395d0c0c9f725970e1a91dca6dd58ac436512900b912e7591b3e351e470607"} Sep 18 17:45:00.828090 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:00.828036 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs" podStartSLOduration=0.828020935 podStartE2EDuration="828.020935ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:45:00.827696487 +0000 UTC m=+1814.080857306" watchObservedRunningTime="2026-09-18 17:45:00.828020935 +0000 UTC m=+1814.081181755" Sep 18 17:45:01.816834 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:01.816796 2560 generic.go:356] "Generic (PLEG): container finished" podID="74e116d6-e2f0-46c9-bbf6-053c0acafa6b" containerID="72c7e81d89141de727616a17d5c78370fb82d1534fc4e2d8cccf50d76781fa4e" exitCode=0 Sep 18 17:45:01.817242 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:01.816885 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs" event={"ID":"74e116d6-e2f0-46c9-bbf6-053c0acafa6b","Type":"ContainerDied","Data":"72c7e81d89141de727616a17d5c78370fb82d1534fc4e2d8cccf50d76781fa4e"} Sep 18 17:45:02.968241 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:02.968212 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs" Sep 18 17:45:03.830258 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:03.830227 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs" Sep 18 17:45:03.830456 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:03.830226 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs" event={"ID":"74e116d6-e2f0-46c9-bbf6-053c0acafa6b","Type":"ContainerDied","Data":"9b395d0c0c9f725970e1a91dca6dd58ac436512900b912e7591b3e351e470607"} Sep 18 17:45:03.830456 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:03.830394 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b395d0c0c9f725970e1a91dca6dd58ac436512900b912e7591b3e351e470607" Sep 18 17:45:08.706723 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:08.706689 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:09.515564 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:09.515530 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:45:20.773496 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:20.773452 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:22.076355 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:22.076314 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:22.944531 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:22.944492 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:23.667381 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:23.667337 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:24.466913 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:24.466874 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:26.473925 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:26.473888 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:27.230855 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.230825 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:27.471053 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.471021 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l"] Sep 18 17:45:27.471510 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.471497 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e116d6-e2f0-46c9-bbf6-053c0acafa6b" containerName="cleanup" Sep 18 17:45:27.471566 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.471512 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e116d6-e2f0-46c9-bbf6-053c0acafa6b" containerName="cleanup" Sep 18 17:45:27.471566 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.471556 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerName="istio-proxy" Sep 18 17:45:27.471566 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.471562 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerName="istio-proxy" Sep 18 17:45:27.471661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.471568 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerName="istio-proxy" Sep 18 17:45:27.471661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.471573 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29db4e1-fcc9-45b2-a4d5-cb8f85fc1068" containerName="istio-proxy" Sep 18 17:45:27.471661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.471589 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerName="istio-proxy" Sep 18 17:45:27.471661 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.471596 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac5040f-d169-4dcb-8a78-4b4e243b87a9" containerName="istio-proxy" Sep 18 17:45:27.471785 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.471667 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f6408dd-e3ae-4a25-b34c-af0a8d3b0183" containerName="istio-proxy" Sep 18 17:45:27.471785 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.471678 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="74e116d6-e2f0-46c9-bbf6-053c0acafa6b" containerName="cleanup" Sep 18 17:45:27.474910 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.474895 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.479944 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.479926 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a8bdd6-openshift-default-dockercfg-wnrcg\"" Sep 18 17:45:27.494329 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.494280 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l"] Sep 18 17:45:27.512686 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.512651 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42493925-5c68-4aad-916d-47abab32b187-istio-token\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.512893 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.512728 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42493925-5c68-4aad-916d-47abab32b187-istio-podinfo\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.512893 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.512782 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npzsd\" (UniqueName: \"kubernetes.io/projected/42493925-5c68-4aad-916d-47abab32b187-kube-api-access-npzsd\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.512893 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.512815 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-credential-socket\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.512893 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.512880 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-workload-socket\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.513119 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.512899 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-istio-envoy\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.513119 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.512936 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42493925-5c68-4aad-916d-47abab32b187-istiod-ca-cert\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.513119 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.512966 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-workload-certs\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.513119 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.512995 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-istio-data\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.524014 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.523984 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:27.613687 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.613653 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npzsd\" (UniqueName: \"kubernetes.io/projected/42493925-5c68-4aad-916d-47abab32b187-kube-api-access-npzsd\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.613687 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.613694 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-credential-socket\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.613965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.613738 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-workload-socket\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.613965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.613758 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-istio-envoy\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.613965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.613787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42493925-5c68-4aad-916d-47abab32b187-istiod-ca-cert\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.613965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.613814 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-workload-certs\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.613965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.613839 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-istio-data\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.613965 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.613911 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42493925-5c68-4aad-916d-47abab32b187-istio-token\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.614327 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.613997 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42493925-5c68-4aad-916d-47abab32b187-istio-podinfo\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.614327 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.614176 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-workload-socket\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.614327 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.614241 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-credential-socket\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.614446 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.614339 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-istio-data\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.614522 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.614504 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-workload-certs\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.614618 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.614599 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42493925-5c68-4aad-916d-47abab32b187-istiod-ca-cert\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.616150 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.616122 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-istio-envoy\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.616587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.616567 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42493925-5c68-4aad-916d-47abab32b187-istio-podinfo\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.616724 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.616705 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:27.622848 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.622695 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42493925-5c68-4aad-916d-47abab32b187-istio-token\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.623375 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.623350 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npzsd\" (UniqueName: \"kubernetes.io/projected/42493925-5c68-4aad-916d-47abab32b187-kube-api-access-npzsd\") pod \"e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:27.785514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:27.785406 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:28.070590 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:28.070564 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l"] Sep 18 17:45:28.075659 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:45:28.075595 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42493925_5c68_4aad_916d_47abab32b187.slice/crio-def2bcccc82ed1ffee7f9a15d043cd12775f2e95a672d86f712e39d22705f0e6 WatchSource:0}: Error finding container def2bcccc82ed1ffee7f9a15d043cd12775f2e95a672d86f712e39d22705f0e6: Status 404 returned error can't find the container with id def2bcccc82ed1ffee7f9a15d043cd12775f2e95a672d86f712e39d22705f0e6 Sep 18 17:45:28.077503 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:28.077463 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:45:28.077593 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:28.077554 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:45:28.077641 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:28.077595 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:45:28.985877 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:28.985839 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" event={"ID":"42493925-5c68-4aad-916d-47abab32b187","Type":"ContainerStarted","Data":"68c6b5d19bca3aee48bf1a80c308c886b768b2568320ec5c0e75891b0fbdad8d"} Sep 18 17:45:28.986381 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:28.985903 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" event={"ID":"42493925-5c68-4aad-916d-47abab32b187","Type":"ContainerStarted","Data":"def2bcccc82ed1ffee7f9a15d043cd12775f2e95a672d86f712e39d22705f0e6"} Sep 18 17:45:29.013653 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:29.013598 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" podStartSLOduration=2.013582227 podStartE2EDuration="2.013582227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:45:29.010908707 +0000 UTC m=+1842.264069551" watchObservedRunningTime="2026-09-18 17:45:29.013582227 +0000 UTC m=+1842.266743044" Sep 18 17:45:29.531125 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:29.531094 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:29.786254 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:29.786165 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:29.791742 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:29.791714 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:29.991922 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:29.991891 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:29.992781 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:29.992761 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:30.362153 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:30.362115 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:31.107337 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:31.107282 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:31.229974 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:31.229937 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:31.325555 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:31.325520 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:32.761985 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:32.761954 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:33.551684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:33.551650 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:34.259686 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:34.259650 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:34.461685 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:34.461650 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:34.743332 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:34.743273 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:34.841587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:34.841544 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:38.431259 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:38.431218 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:45:49.117463 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:49.117428 2560 generic.go:356] "Generic (PLEG): container finished" podID="42493925-5c68-4aad-916d-47abab32b187" containerID="68c6b5d19bca3aee48bf1a80c308c886b768b2568320ec5c0e75891b0fbdad8d" exitCode=0 Sep 18 17:45:49.117912 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:49.117506 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" event={"ID":"42493925-5c68-4aad-916d-47abab32b187","Type":"ContainerDied","Data":"68c6b5d19bca3aee48bf1a80c308c886b768b2568320ec5c0e75891b0fbdad8d"} Sep 18 17:45:49.117966 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:49.117923 2560 scope.go:117] "RemoveContainer" containerID="68c6b5d19bca3aee48bf1a80c308c886b768b2568320ec5c0e75891b0fbdad8d" Sep 18 17:45:49.119066 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:49.119033 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:45:49.119164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:49.119106 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:45:49.119164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:49.119142 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:45:50.126702 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:50.126658 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" event={"ID":"42493925-5c68-4aad-916d-47abab32b187","Type":"ContainerStarted","Data":"d84a38aca9b34b92502be35dd7d9a06f31439d195d06cf223ab3c94e611a98a1"} Sep 18 17:45:50.785857 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:50.785818 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:50.786047 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:50.786004 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 18 17:45:50.786094 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:50.786067 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" podUID="42493925-5c68-4aad-916d-47abab32b187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 18 17:45:51.786487 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:51.786446 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 18 17:45:51.786910 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:51.786521 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" podUID="42493925-5c68-4aad-916d-47abab32b187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 18 17:45:52.786175 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:52.786136 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 18 17:45:52.786400 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:52.786212 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" podUID="42493925-5c68-4aad-916d-47abab32b187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 18 17:45:53.785983 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:53.785945 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 18 17:45:53.786414 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:53.786002 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" podUID="42493925-5c68-4aad-916d-47abab32b187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 18 17:45:54.785916 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:54.785872 2560 patch_prober.go:28] interesting pod/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Sep 18 17:45:54.786110 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:54.785948 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" podUID="42493925-5c68-4aad-916d-47abab32b187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Sep 18 17:45:55.790106 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:55.790074 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:55.790525 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:55.790422 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:55.791217 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:55.791201 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:45:56.350575 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:56.350542 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:57.596680 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:57.596645 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:57.841377 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:57.841343 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:58.514521 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:58.514484 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:45:59.332091 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:45:59.332052 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:01.061912 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:01.061790 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:07.534068 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:07.533964 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:24.291580 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:24.291520 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:28.411921 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:28.411886 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:46:41.815851 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:41.815809 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:46:48.126525 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.126494 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:48.397230 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.397192 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws"] Sep 18 17:46:48.401540 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.401518 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.404568 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.404545 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-88b9e1-openshift-default-dockercfg-mjjwq\"" Sep 18 17:46:48.415261 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.415234 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws"] Sep 18 17:46:48.424766 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.424729 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:48.449676 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.449634 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-data\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.449853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.449697 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-workload-socket\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.449853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.449747 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-envoy\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.449853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.449831 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-token\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.450082 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.449881 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.450082 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.449919 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-podinfo\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.450082 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.449961 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-workload-certs\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.450082 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.450026 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmh82\" (UniqueName: \"kubernetes.io/projected/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-kube-api-access-lmh82\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.450332 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.450078 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-credential-socket\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.551475 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.551430 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-workload-socket\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.551684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.551483 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-envoy\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.551684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.551518 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-token\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.551684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.551538 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.551684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.551568 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-podinfo\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.551684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.551594 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-workload-certs\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.551684 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.551632 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmh82\" (UniqueName: \"kubernetes.io/projected/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-kube-api-access-lmh82\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.551993 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.551744 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-credential-socket\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.551993 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.551941 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-workload-socket\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.552110 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.552059 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-workload-certs\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.552184 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.552166 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-data\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.552247 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.552222 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-credential-socket\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.552399 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.552370 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.552544 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.552474 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-data\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.554026 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.553995 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-envoy\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.554316 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.554275 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-podinfo\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.560224 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.560195 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-token\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.560345 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.560201 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmh82\" (UniqueName: \"kubernetes.io/projected/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-kube-api-access-lmh82\") pod \"e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.713677 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.713568 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:48.758194 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:48.758155 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:49.010034 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:49.009954 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws"] Sep 18 17:46:49.014190 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:46:49.014161 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d9e32_57db_4e74_a2bd_99ebb26d9bfd.slice/crio-b2b22bf5872b83c2c953efc506b4964a49066d0a1e4bf6d90e10a94d9ea65ce2 WatchSource:0}: Error finding container b2b22bf5872b83c2c953efc506b4964a49066d0a1e4bf6d90e10a94d9ea65ce2: Status 404 returned error can't find the container with id b2b22bf5872b83c2c953efc506b4964a49066d0a1e4bf6d90e10a94d9ea65ce2 Sep 18 17:46:49.016578 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:49.016543 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:46:49.016648 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:49.016615 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:46:49.016648 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:49.016643 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:46:49.477970 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:49.477938 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" event={"ID":"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd","Type":"ContainerStarted","Data":"98bec8edcc7821070e743be9406ae3fa5c6a7cfb9e9a018c1a593027c4894305"} Sep 18 17:46:49.478394 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:49.477997 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" event={"ID":"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd","Type":"ContainerStarted","Data":"b2b22bf5872b83c2c953efc506b4964a49066d0a1e4bf6d90e10a94d9ea65ce2"} Sep 18 17:46:49.501127 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:49.501060 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" podStartSLOduration=1.501042859 podStartE2EDuration="1.501042859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:46:49.498890519 +0000 UTC m=+1922.752051355" watchObservedRunningTime="2026-09-18 17:46:49.501042859 +0000 UTC m=+1922.754203676" Sep 18 17:46:49.713694 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:49.713661 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:50.719419 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:50.719387 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:50.857181 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:50.857151 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:51.449220 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:51.449188 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:51.490002 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:51.489974 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:51.490863 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:51.490845 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:46:51.581571 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:51.581541 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:51.944186 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:51.944155 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:52.944194 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:52.944160 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:53.742183 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:53.742150 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:54.495194 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:54.495148 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:55.268354 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:55.268276 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:55.371642 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:55.371613 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:55.655578 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:55.655546 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:55.966377 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:55.966278 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:46:56.065068 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:46:56.065031 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:02.079127 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:02.079083 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:47:13.623791 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:13.623755 2560 generic.go:356] "Generic (PLEG): container finished" podID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerID="98bec8edcc7821070e743be9406ae3fa5c6a7cfb9e9a018c1a593027c4894305" exitCode=0 Sep 18 17:47:13.624282 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:13.623825 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" event={"ID":"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd","Type":"ContainerDied","Data":"98bec8edcc7821070e743be9406ae3fa5c6a7cfb9e9a018c1a593027c4894305"} Sep 18 17:47:13.624400 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:13.624320 2560 scope.go:117] "RemoveContainer" containerID="98bec8edcc7821070e743be9406ae3fa5c6a7cfb9e9a018c1a593027c4894305" Sep 18 17:47:13.625563 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:13.625506 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:47:13.625638 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:13.625581 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:47:13.625638 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:13.625614 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 18 17:47:14.631795 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:14.631760 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" event={"ID":"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd","Type":"ContainerStarted","Data":"b69164933f33ddb73ea579acc60eeb33645ae725e59e6c0a1b367f1765d02cb8"} Sep 18 17:47:14.714516 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:14.714473 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:47:14.714855 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:14.714828 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 18 17:47:14.714945 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:14.714887 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 18 17:47:15.714594 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:15.714548 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 18 17:47:15.715021 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:15.714621 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 18 17:47:16.714466 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:16.714433 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 18 17:47:16.714649 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:16.714500 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 18 17:47:17.714733 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:17.714690 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 18 17:47:17.715153 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:17.714753 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 18 17:47:18.148176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:18.148135 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:18.714956 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:18.714921 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 18 17:47:18.715383 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:18.714977 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 18 17:47:19.158124 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:19.158079 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:19.714840 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:19.714801 2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 18 17:47:19.715046 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:19.714859 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 18 17:47:19.923006 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:19.922946 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:20.718097 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:20.718062 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:47:20.718611 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:20.718457 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:47:20.719230 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:20.719215 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:47:20.898448 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:20.898418 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:21.510220 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:21.510188 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:21.640942 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:21.640911 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:22.605199 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:22.605160 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:23.440172 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:23.440139 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:24.334393 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:24.334361 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:25.022681 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:25.022647 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:25.838584 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:25.838552 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:26.638044 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:26.637926 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:32.826366 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:32.826255 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:33.102760 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:33.102675 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:33.201653 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:33.201622 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:33.449915 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:33.449879 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:33.612500 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:33.612465 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:33.730642 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:33.730556 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:33.856480 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:33.856446 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:33.934452 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:33.934416 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:34.259250 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:34.259216 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:34.331825 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:34.331792 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:35.194983 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:35.194941 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:35.936219 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:35.936189 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:36.672656 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:36.672623 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:37.335606 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:37.335569 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:38.204606 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:38.204571 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:47:40.338947 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:40.338910 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:40.403559 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:40.403531 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:40.594236 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:40.594155 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws"] Sep 18 17:47:40.594460 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:40.594436 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerName="istio-proxy" containerID="cri-o://b69164933f33ddb73ea579acc60eeb33645ae725e59e6c0a1b367f1765d02cb8" gracePeriod=30 Sep 18 17:47:40.618652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:40.618622 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:44.205926 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:44.205897 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:45.028538 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.028498 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:45.791587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.791557 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:45.816498 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.816471 2560 generic.go:356] "Generic (PLEG): container finished" podID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerID="b69164933f33ddb73ea579acc60eeb33645ae725e59e6c0a1b367f1765d02cb8" exitCode=0 Sep 18 17:47:45.816715 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.816545 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" event={"ID":"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd","Type":"ContainerDied","Data":"b69164933f33ddb73ea579acc60eeb33645ae725e59e6c0a1b367f1765d02cb8"} Sep 18 17:47:45.816715 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.816605 2560 scope.go:117] "RemoveContainer" containerID="98bec8edcc7821070e743be9406ae3fa5c6a7cfb9e9a018c1a593027c4894305" Sep 18 17:47:45.854676 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.854597 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:47:45.884255 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.884224 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-data\") pod \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " Sep 18 17:47:45.884853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.884284 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-workload-certs\") pod \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " Sep 18 17:47:45.884853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.884350 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-workload-socket\") pod \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " Sep 18 17:47:45.884853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.884399 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmh82\" (UniqueName: \"kubernetes.io/projected/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-kube-api-access-lmh82\") pod \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " Sep 18 17:47:45.884853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.884431 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-podinfo\") pod \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " Sep 18 17:47:45.884853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.884490 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-credential-socket\") pod \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " Sep 18 17:47:45.884853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.884564 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-envoy\") pod \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " Sep 18 17:47:45.884853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.884599 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istiod-ca-cert\") pod \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " Sep 18 17:47:45.884853 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.884640 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-token\") pod \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\" (UID: \"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd\") " Sep 18 17:47:45.885304 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.885184 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" (UID: "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:47:45.885776 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.885720 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" (UID: "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:47:45.885880 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.885817 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-data" (OuterVolumeSpecName: "istio-data") pod "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" (UID: "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:47:45.886022 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.885977 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" (UID: "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:47:45.886192 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.886170 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" (UID: "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:47:45.887833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.887802 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" (UID: "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:47:45.887969 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.887950 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" (UID: "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:47:45.888377 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.888351 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-token" (OuterVolumeSpecName: "istio-token") pod "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" (UID: "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:47:45.888488 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.888463 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-kube-api-access-lmh82" (OuterVolumeSpecName: "kube-api-access-lmh82") pod "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" (UID: "e94d9e32-57db-4e74-a2bd-99ebb26d9bfd"). InnerVolumeSpecName "kube-api-access-lmh82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:47:45.986034 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.986004 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:47:45.986034 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.986033 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:47:45.986034 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.986045 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:47:45.986276 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.986057 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmh82\" (UniqueName: \"kubernetes.io/projected/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-kube-api-access-lmh82\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:47:45.986276 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.986069 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:47:45.986276 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.986079 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:47:45.986276 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.986090 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:47:45.986276 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.986101 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:47:45.986276 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:45.986113 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:47:46.654752 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:46.654723 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:46.823928 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:46.823898 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" event={"ID":"e94d9e32-57db-4e74-a2bd-99ebb26d9bfd","Type":"ContainerDied","Data":"b2b22bf5872b83c2c953efc506b4964a49066d0a1e4bf6d90e10a94d9ea65ce2"} Sep 18 17:47:46.824336 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:46.823950 2560 scope.go:117] "RemoveContainer" containerID="b69164933f33ddb73ea579acc60eeb33645ae725e59e6c0a1b367f1765d02cb8" Sep 18 17:47:46.824336 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:46.823905 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws" Sep 18 17:47:46.853716 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:46.853681 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws"] Sep 18 17:47:46.858020 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:46.857997 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-88b9e1-openshift-default-6ffcd7599d-rjmws"] Sep 18 17:47:46.864974 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:46.864948 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" path="/var/lib/kubelet/pods/e94d9e32-57db-4e74-a2bd-99ebb26d9bfd/volumes" Sep 18 17:47:47.823505 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:47.823457 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:48.623579 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:48.623544 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:57.109515 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:57.109477 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:47:59.723746 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:59.723712 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:59.853562 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:59.853524 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:47:59.935074 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:47:59.935042 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:00.032941 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:00.032859 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:00.120026 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:00.119984 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:00.447497 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:00.447458 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:00.622682 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:00.622646 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:01.822185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:01.822150 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:02.891341 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:02.891285 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:03.590881 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:03.590851 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:05.588407 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:05.588375 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:11.551664 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:11.551628 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:48:42.929516 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:42.929476 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:43.213218 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:43.213132 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:49.364860 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:49.364822 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:55.568161 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:55.568126 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:55.678645 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:55.678609 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:55.874094 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:55.874004 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:57.558166 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:57.558136 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:58.536800 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:58.536769 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:59.457349 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:59.457315 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:48:59.984692 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:48:59.984661 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:05.551208 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:05.551173 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:05.625675 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:05.625644 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:07.077528 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:07.077497 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:07.175282 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:07.175251 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:07.264973 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:07.264891 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:07.564777 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:07.564746 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:07.656617 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:07.656583 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:08.351420 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:08.351387 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:09.220217 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:09.220183 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:10.597265 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:10.597217 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:49:14.199202 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:14.199171 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:14.300528 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:14.300497 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:14.465064 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:14.464969 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l"] Sep 18 17:49:14.465271 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:14.465246 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" podUID="42493925-5c68-4aad-916d-47abab32b187" containerName="istio-proxy" containerID="cri-o://d84a38aca9b34b92502be35dd7d9a06f31439d195d06cf223ab3c94e611a98a1" gracePeriod=30 Sep 18 17:49:14.491563 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:14.491533 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:15.025538 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:15.025506 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:16.272604 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:16.272573 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:17.077047 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:17.077012 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:17.767865 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:17.767835 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:19.729111 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.729037 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:49:19.782986 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.782953 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npzsd\" (UniqueName: \"kubernetes.io/projected/42493925-5c68-4aad-916d-47abab32b187-kube-api-access-npzsd\") pod \"42493925-5c68-4aad-916d-47abab32b187\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " Sep 18 17:49:19.783204 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783028 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42493925-5c68-4aad-916d-47abab32b187-istio-token\") pod \"42493925-5c68-4aad-916d-47abab32b187\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " Sep 18 17:49:19.783204 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783070 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-istio-data\") pod \"42493925-5c68-4aad-916d-47abab32b187\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " Sep 18 17:49:19.783204 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783104 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42493925-5c68-4aad-916d-47abab32b187-istiod-ca-cert\") pod \"42493925-5c68-4aad-916d-47abab32b187\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " Sep 18 17:49:19.783204 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783136 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-credential-socket\") pod \"42493925-5c68-4aad-916d-47abab32b187\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " Sep 18 17:49:19.783204 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783170 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-workload-certs\") pod \"42493925-5c68-4aad-916d-47abab32b187\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " Sep 18 17:49:19.783510 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783248 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42493925-5c68-4aad-916d-47abab32b187-istio-podinfo\") pod \"42493925-5c68-4aad-916d-47abab32b187\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " Sep 18 17:49:19.783510 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783319 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-istio-envoy\") pod \"42493925-5c68-4aad-916d-47abab32b187\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " Sep 18 17:49:19.783510 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783361 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-workload-socket\") pod \"42493925-5c68-4aad-916d-47abab32b187\" (UID: \"42493925-5c68-4aad-916d-47abab32b187\") " Sep 18 17:49:19.783510 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783421 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-istio-data" (OuterVolumeSpecName: "istio-data") pod "42493925-5c68-4aad-916d-47abab32b187" (UID: "42493925-5c68-4aad-916d-47abab32b187"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:19.783719 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783677 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42493925-5c68-4aad-916d-47abab32b187-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42493925-5c68-4aad-916d-47abab32b187" (UID: "42493925-5c68-4aad-916d-47abab32b187"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:49:19.783774 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783760 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:19.783832 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783764 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42493925-5c68-4aad-916d-47abab32b187" (UID: "42493925-5c68-4aad-916d-47abab32b187"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:19.783832 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783824 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42493925-5c68-4aad-916d-47abab32b187" (UID: "42493925-5c68-4aad-916d-47abab32b187"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:19.783987 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.783969 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42493925-5c68-4aad-916d-47abab32b187" (UID: "42493925-5c68-4aad-916d-47abab32b187"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:19.785564 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.785532 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42493925-5c68-4aad-916d-47abab32b187-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42493925-5c68-4aad-916d-47abab32b187" (UID: "42493925-5c68-4aad-916d-47abab32b187"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:49:19.785686 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.785597 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42493925-5c68-4aad-916d-47abab32b187-kube-api-access-npzsd" (OuterVolumeSpecName: "kube-api-access-npzsd") pod "42493925-5c68-4aad-916d-47abab32b187" (UID: "42493925-5c68-4aad-916d-47abab32b187"). InnerVolumeSpecName "kube-api-access-npzsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:49:19.785839 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.785822 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42493925-5c68-4aad-916d-47abab32b187-istio-token" (OuterVolumeSpecName: "istio-token") pod "42493925-5c68-4aad-916d-47abab32b187" (UID: "42493925-5c68-4aad-916d-47abab32b187"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:49:19.785886 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.785833 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42493925-5c68-4aad-916d-47abab32b187" (UID: "42493925-5c68-4aad-916d-47abab32b187"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:19.884842 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.884798 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npzsd\" (UniqueName: \"kubernetes.io/projected/42493925-5c68-4aad-916d-47abab32b187-kube-api-access-npzsd\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:19.884842 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.884834 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42493925-5c68-4aad-916d-47abab32b187-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:19.884842 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.884847 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42493925-5c68-4aad-916d-47abab32b187-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:19.885149 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.884856 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:19.885149 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.884864 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:19.885149 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.884875 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42493925-5c68-4aad-916d-47abab32b187-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:19.885149 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.884884 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:19.885149 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:19.884892 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42493925-5c68-4aad-916d-47abab32b187-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:20.332973 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.332941 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:20.362904 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.362869 2560 generic.go:356] "Generic (PLEG): container finished" podID="42493925-5c68-4aad-916d-47abab32b187" containerID="d84a38aca9b34b92502be35dd7d9a06f31439d195d06cf223ab3c94e611a98a1" exitCode=0 Sep 18 17:49:20.363116 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.362923 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" Sep 18 17:49:20.363116 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.362955 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" event={"ID":"42493925-5c68-4aad-916d-47abab32b187","Type":"ContainerDied","Data":"d84a38aca9b34b92502be35dd7d9a06f31439d195d06cf223ab3c94e611a98a1"} Sep 18 17:49:20.363116 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.363021 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l" event={"ID":"42493925-5c68-4aad-916d-47abab32b187","Type":"ContainerDied","Data":"def2bcccc82ed1ffee7f9a15d043cd12775f2e95a672d86f712e39d22705f0e6"} Sep 18 17:49:20.363116 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.363050 2560 scope.go:117] "RemoveContainer" containerID="d84a38aca9b34b92502be35dd7d9a06f31439d195d06cf223ab3c94e611a98a1" Sep 18 17:49:20.374214 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.374186 2560 scope.go:117] "RemoveContainer" containerID="68c6b5d19bca3aee48bf1a80c308c886b768b2568320ec5c0e75891b0fbdad8d" Sep 18 17:49:20.384922 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.384891 2560 scope.go:117] "RemoveContainer" containerID="d84a38aca9b34b92502be35dd7d9a06f31439d195d06cf223ab3c94e611a98a1" Sep 18 17:49:20.385398 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:49:20.385330 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84a38aca9b34b92502be35dd7d9a06f31439d195d06cf223ab3c94e611a98a1\": container with ID starting with d84a38aca9b34b92502be35dd7d9a06f31439d195d06cf223ab3c94e611a98a1 not found: ID does not exist" containerID="d84a38aca9b34b92502be35dd7d9a06f31439d195d06cf223ab3c94e611a98a1" Sep 18 17:49:20.385398 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.385371 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84a38aca9b34b92502be35dd7d9a06f31439d195d06cf223ab3c94e611a98a1"} err="failed to get container status \"d84a38aca9b34b92502be35dd7d9a06f31439d195d06cf223ab3c94e611a98a1\": rpc error: code = NotFound desc = could not find container \"d84a38aca9b34b92502be35dd7d9a06f31439d195d06cf223ab3c94e611a98a1\": container with ID starting with d84a38aca9b34b92502be35dd7d9a06f31439d195d06cf223ab3c94e611a98a1 not found: ID does not exist" Sep 18 17:49:20.385398 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.385397 2560 scope.go:117] "RemoveContainer" containerID="68c6b5d19bca3aee48bf1a80c308c886b768b2568320ec5c0e75891b0fbdad8d" Sep 18 17:49:20.385828 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:49:20.385807 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c6b5d19bca3aee48bf1a80c308c886b768b2568320ec5c0e75891b0fbdad8d\": container with ID starting with 68c6b5d19bca3aee48bf1a80c308c886b768b2568320ec5c0e75891b0fbdad8d not found: ID does not exist" containerID="68c6b5d19bca3aee48bf1a80c308c886b768b2568320ec5c0e75891b0fbdad8d" Sep 18 17:49:20.385914 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.385833 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c6b5d19bca3aee48bf1a80c308c886b768b2568320ec5c0e75891b0fbdad8d"} err="failed to get container status \"68c6b5d19bca3aee48bf1a80c308c886b768b2568320ec5c0e75891b0fbdad8d\": rpc error: code = NotFound desc = could not find container \"68c6b5d19bca3aee48bf1a80c308c886b768b2568320ec5c0e75891b0fbdad8d\": container with ID starting with 68c6b5d19bca3aee48bf1a80c308c886b768b2568320ec5c0e75891b0fbdad8d not found: ID does not exist" Sep 18 17:49:20.387984 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.387940 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l"] Sep 18 17:49:20.391947 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.391920 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a8bdd6-openshift-default-5ccdfcffb6-p8l7l"] Sep 18 17:49:20.865164 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:20.865131 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42493925-5c68-4aad-916d-47abab32b187" path="/var/lib/kubelet/pods/42493925-5c68-4aad-916d-47abab32b187/volumes" Sep 18 17:49:25.381620 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.381586 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv"] Sep 18 17:49:25.382106 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.381790 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" podUID="972da268-f047-46c4-a566-ad813b1a0687" containerName="maas-api" containerID="cri-o://d59ad26ffccf46c2b1fd4f224d28b2eefe74406507076e5d4c262810e7b2fb0e" gracePeriod=30 Sep 18 17:49:25.419074 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.419039 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs"] Sep 18 17:49:25.426404 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.426377 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-eeab6e-29829225-gbdfs"] Sep 18 17:49:25.434830 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.434797 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:25.515854 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.515812 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:25.615193 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.615162 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:25.642067 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.642012 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:49:25.726833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.726803 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:25.750059 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.750021 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ppvl\" (UniqueName: \"kubernetes.io/projected/972da268-f047-46c4-a566-ad813b1a0687-kube-api-access-7ppvl\") pod \"972da268-f047-46c4-a566-ad813b1a0687\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " Sep 18 17:49:25.750250 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.750097 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972da268-f047-46c4-a566-ad813b1a0687-openshift-service-ca\") pod \"972da268-f047-46c4-a566-ad813b1a0687\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " Sep 18 17:49:25.750250 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.750188 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-metrics-certs\") pod \"972da268-f047-46c4-a566-ad813b1a0687\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " Sep 18 17:49:25.750402 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.750258 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-maas-api-tls\") pod \"972da268-f047-46c4-a566-ad813b1a0687\" (UID: \"972da268-f047-46c4-a566-ad813b1a0687\") " Sep 18 17:49:25.750652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.750620 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972da268-f047-46c4-a566-ad813b1a0687-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "972da268-f047-46c4-a566-ad813b1a0687" (UID: "972da268-f047-46c4-a566-ad813b1a0687"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:49:25.752592 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.752562 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "972da268-f047-46c4-a566-ad813b1a0687" (UID: "972da268-f047-46c4-a566-ad813b1a0687"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:49:25.752878 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.752854 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972da268-f047-46c4-a566-ad813b1a0687-kube-api-access-7ppvl" (OuterVolumeSpecName: "kube-api-access-7ppvl") pod "972da268-f047-46c4-a566-ad813b1a0687" (UID: "972da268-f047-46c4-a566-ad813b1a0687"). InnerVolumeSpecName "kube-api-access-7ppvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:49:25.753024 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.753007 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "972da268-f047-46c4-a566-ad813b1a0687" (UID: "972da268-f047-46c4-a566-ad813b1a0687"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:49:25.852346 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.852288 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ppvl\" (UniqueName: \"kubernetes.io/projected/972da268-f047-46c4-a566-ad813b1a0687-kube-api-access-7ppvl\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:25.852346 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.852342 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/972da268-f047-46c4-a566-ad813b1a0687-openshift-service-ca\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:25.852556 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.852357 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-metrics-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:25.852556 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:25.852370 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/972da268-f047-46c4-a566-ad813b1a0687-maas-api-tls\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:26.044423 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:26.044389 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:26.154457 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:26.154422 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:26.401154 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:26.401120 2560 generic.go:356] "Generic (PLEG): container finished" podID="972da268-f047-46c4-a566-ad813b1a0687" containerID="d59ad26ffccf46c2b1fd4f224d28b2eefe74406507076e5d4c262810e7b2fb0e" exitCode=0 Sep 18 17:49:26.401652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:26.401161 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" Sep 18 17:49:26.401652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:26.401198 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" event={"ID":"972da268-f047-46c4-a566-ad813b1a0687","Type":"ContainerDied","Data":"d59ad26ffccf46c2b1fd4f224d28b2eefe74406507076e5d4c262810e7b2fb0e"} Sep 18 17:49:26.401652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:26.401247 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv" event={"ID":"972da268-f047-46c4-a566-ad813b1a0687","Type":"ContainerDied","Data":"61015f3846de31ebf8c0136556135dd63b72aad590651d5ab6b7bc5d58a83dd7"} Sep 18 17:49:26.401652 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:26.401270 2560 scope.go:117] "RemoveContainer" containerID="d59ad26ffccf46c2b1fd4f224d28b2eefe74406507076e5d4c262810e7b2fb0e" Sep 18 17:49:26.412085 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:26.412064 2560 scope.go:117] "RemoveContainer" containerID="d59ad26ffccf46c2b1fd4f224d28b2eefe74406507076e5d4c262810e7b2fb0e" Sep 18 17:49:26.412395 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:49:26.412375 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59ad26ffccf46c2b1fd4f224d28b2eefe74406507076e5d4c262810e7b2fb0e\": container with ID starting with d59ad26ffccf46c2b1fd4f224d28b2eefe74406507076e5d4c262810e7b2fb0e not found: ID does not exist" containerID="d59ad26ffccf46c2b1fd4f224d28b2eefe74406507076e5d4c262810e7b2fb0e" Sep 18 17:49:26.412464 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:26.412403 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59ad26ffccf46c2b1fd4f224d28b2eefe74406507076e5d4c262810e7b2fb0e"} err="failed to get container status \"d59ad26ffccf46c2b1fd4f224d28b2eefe74406507076e5d4c262810e7b2fb0e\": rpc error: code = NotFound desc = could not find container \"d59ad26ffccf46c2b1fd4f224d28b2eefe74406507076e5d4c262810e7b2fb0e\": container with ID starting with d59ad26ffccf46c2b1fd4f224d28b2eefe74406507076e5d4c262810e7b2fb0e not found: ID does not exist" Sep 18 17:49:26.435065 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:26.435028 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv"] Sep 18 17:49:26.438774 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:26.438750 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-eeab6e-7b59ff846-zmhzv"] Sep 18 17:49:26.864926 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:26.864892 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74e116d6-e2f0-46c9-bbf6-053c0acafa6b" path="/var/lib/kubelet/pods/74e116d6-e2f0-46c9-bbf6-053c0acafa6b/volumes" Sep 18 17:49:26.865279 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:26.865266 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="972da268-f047-46c4-a566-ad813b1a0687" path="/var/lib/kubelet/pods/972da268-f047-46c4-a566-ad813b1a0687/volumes" Sep 18 17:49:27.247321 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:27.247277 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:28.615846 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:28.615808 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:49:32.344513 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:32.344474 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:32.418137 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:32.418104 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:32.611842 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:32.611749 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9"] Sep 18 17:49:32.612053 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:32.612026 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerName="istio-proxy" containerID="cri-o://6fbc910e67e3197727180876972cb5182787b2de24c55caac63a09234b4b4c65" gracePeriod=30 Sep 18 17:49:32.625978 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:32.625944 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:33.730106 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:33.730050 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:34.741454 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:34.741419 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:37.864722 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.864699 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:49:37.987713 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.987682 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istiod-ca-cert\") pod \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " Sep 18 17:49:37.987713 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.987716 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-workload-socket\") pod \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " Sep 18 17:49:37.987986 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.987735 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwzn9\" (UniqueName: \"kubernetes.io/projected/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-kube-api-access-nwzn9\") pod \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " Sep 18 17:49:37.987986 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.987758 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-workload-certs\") pod \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " Sep 18 17:49:37.988103 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.987990 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-envoy\") pod \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " Sep 18 17:49:37.988103 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.988054 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-podinfo\") pod \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " Sep 18 17:49:37.988103 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.988052 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" (UID: "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:37.988103 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.988069 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" (UID: "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:37.988103 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.988089 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-token\") pod \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " Sep 18 17:49:37.988426 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.988136 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-credential-socket\") pod \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " Sep 18 17:49:37.988426 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.988171 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" (UID: "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:49:37.988426 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.988211 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-data\") pod \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\" (UID: \"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c\") " Sep 18 17:49:37.988577 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.988498 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" (UID: "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:37.988577 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.988564 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-data" (OuterVolumeSpecName: "istio-data") pod "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" (UID: "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:37.988696 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.988630 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:37.988696 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.988650 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:37.988696 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.988663 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:37.988696 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.988675 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:37.990228 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.990204 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" (UID: "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:37.990357 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.990249 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" (UID: "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:49:37.990623 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.990602 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-token" (OuterVolumeSpecName: "istio-token") pod "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" (UID: "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:49:37.990679 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:37.990634 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-kube-api-access-nwzn9" (OuterVolumeSpecName: "kube-api-access-nwzn9") pod "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" (UID: "a33da0e8-b61f-4bca-aa2a-afc1cdb5628c"). InnerVolumeSpecName "kube-api-access-nwzn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:49:38.089883 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.089834 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:38.089883 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.089880 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwzn9\" (UniqueName: \"kubernetes.io/projected/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-kube-api-access-nwzn9\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:38.089883 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.089895 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:38.090127 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.089906 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:38.090127 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.089917 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:38.468858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.468824 2560 generic.go:356] "Generic (PLEG): container finished" podID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerID="6fbc910e67e3197727180876972cb5182787b2de24c55caac63a09234b4b4c65" exitCode=0 Sep 18 17:49:38.469048 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.468877 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" Sep 18 17:49:38.469048 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.468885 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" event={"ID":"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c","Type":"ContainerDied","Data":"6fbc910e67e3197727180876972cb5182787b2de24c55caac63a09234b4b4c65"} Sep 18 17:49:38.469048 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.468927 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9" event={"ID":"a33da0e8-b61f-4bca-aa2a-afc1cdb5628c","Type":"ContainerDied","Data":"3dde24efc2bef95e36899be7802e03575e2891ec6dd172284b60e63ac0e70498"} Sep 18 17:49:38.469048 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.468950 2560 scope.go:117] "RemoveContainer" containerID="6fbc910e67e3197727180876972cb5182787b2de24c55caac63a09234b4b4c65" Sep 18 17:49:38.478923 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.478904 2560 scope.go:117] "RemoveContainer" containerID="10daff737d529a3a15178c24d0c763fbc94277cda82b2a83ca4c131da55ad2ea" Sep 18 17:49:38.487055 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.487040 2560 scope.go:117] "RemoveContainer" containerID="6fbc910e67e3197727180876972cb5182787b2de24c55caac63a09234b4b4c65" Sep 18 17:49:38.487342 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:49:38.487320 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbc910e67e3197727180876972cb5182787b2de24c55caac63a09234b4b4c65\": container with ID starting with 6fbc910e67e3197727180876972cb5182787b2de24c55caac63a09234b4b4c65 not found: ID does not exist" containerID="6fbc910e67e3197727180876972cb5182787b2de24c55caac63a09234b4b4c65" Sep 18 17:49:38.487396 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.487351 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbc910e67e3197727180876972cb5182787b2de24c55caac63a09234b4b4c65"} err="failed to get container status \"6fbc910e67e3197727180876972cb5182787b2de24c55caac63a09234b4b4c65\": rpc error: code = NotFound desc = could not find container \"6fbc910e67e3197727180876972cb5182787b2de24c55caac63a09234b4b4c65\": container with ID starting with 6fbc910e67e3197727180876972cb5182787b2de24c55caac63a09234b4b4c65 not found: ID does not exist" Sep 18 17:49:38.487396 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.487369 2560 scope.go:117] "RemoveContainer" containerID="10daff737d529a3a15178c24d0c763fbc94277cda82b2a83ca4c131da55ad2ea" Sep 18 17:49:38.487586 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:49:38.487569 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10daff737d529a3a15178c24d0c763fbc94277cda82b2a83ca4c131da55ad2ea\": container with ID starting with 10daff737d529a3a15178c24d0c763fbc94277cda82b2a83ca4c131da55ad2ea not found: ID does not exist" containerID="10daff737d529a3a15178c24d0c763fbc94277cda82b2a83ca4c131da55ad2ea" Sep 18 17:49:38.487627 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.487590 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10daff737d529a3a15178c24d0c763fbc94277cda82b2a83ca4c131da55ad2ea"} err="failed to get container status \"10daff737d529a3a15178c24d0c763fbc94277cda82b2a83ca4c131da55ad2ea\": rpc error: code = NotFound desc = could not find container \"10daff737d529a3a15178c24d0c763fbc94277cda82b2a83ca4c131da55ad2ea\": container with ID starting with 10daff737d529a3a15178c24d0c763fbc94277cda82b2a83ca4c131da55ad2ea not found: ID does not exist" Sep 18 17:49:38.494003 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.493975 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9"] Sep 18 17:49:38.496581 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.496561 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-eeab6e-openshift-default-6bdc677b8-mvdd9"] Sep 18 17:49:38.866455 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:38.866379 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" path="/var/lib/kubelet/pods/a33da0e8-b61f-4bca-aa2a-afc1cdb5628c/volumes" Sep 18 17:49:43.613790 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.613752 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp"] Sep 18 17:49:43.614277 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.614020 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" podUID="587f3ae3-56fa-4ece-8580-f9787cdc0db4" containerName="maas-api" containerID="cri-o://8e46c9e3632ec2a158a2d1401a62c8ddf61ced4a769594148f1d10e82c93a452" gracePeriod=30 Sep 18 17:49:43.650603 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.650568 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:43.730052 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.730018 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:43.803444 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.803413 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:43.875217 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.875148 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:49:43.877496 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.877448 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:43.941440 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.941407 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-metrics-certs\") pod \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " Sep 18 17:49:43.941649 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.941451 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-maas-api-tls\") pod \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " Sep 18 17:49:43.941649 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.941480 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587f3ae3-56fa-4ece-8580-f9787cdc0db4-openshift-service-ca\") pod \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " Sep 18 17:49:43.941649 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.941519 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft4pv\" (UniqueName: \"kubernetes.io/projected/587f3ae3-56fa-4ece-8580-f9787cdc0db4-kube-api-access-ft4pv\") pod \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\" (UID: \"587f3ae3-56fa-4ece-8580-f9787cdc0db4\") " Sep 18 17:49:43.941926 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.941897 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587f3ae3-56fa-4ece-8580-f9787cdc0db4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "587f3ae3-56fa-4ece-8580-f9787cdc0db4" (UID: "587f3ae3-56fa-4ece-8580-f9787cdc0db4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:49:43.943869 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.943843 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "587f3ae3-56fa-4ece-8580-f9787cdc0db4" (UID: "587f3ae3-56fa-4ece-8580-f9787cdc0db4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:49:43.944002 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.943932 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "587f3ae3-56fa-4ece-8580-f9787cdc0db4" (UID: "587f3ae3-56fa-4ece-8580-f9787cdc0db4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 18 17:49:43.944185 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:43.944160 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587f3ae3-56fa-4ece-8580-f9787cdc0db4-kube-api-access-ft4pv" (OuterVolumeSpecName: "kube-api-access-ft4pv") pod "587f3ae3-56fa-4ece-8580-f9787cdc0db4" (UID: "587f3ae3-56fa-4ece-8580-f9787cdc0db4"). InnerVolumeSpecName "kube-api-access-ft4pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:49:44.042945 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.042909 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-metrics-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:44.042945 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.042939 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/587f3ae3-56fa-4ece-8580-f9787cdc0db4-maas-api-tls\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:44.042945 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.042949 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/587f3ae3-56fa-4ece-8580-f9787cdc0db4-openshift-service-ca\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:44.042945 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.042959 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft4pv\" (UniqueName: \"kubernetes.io/projected/587f3ae3-56fa-4ece-8580-f9787cdc0db4-kube-api-access-ft4pv\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:44.220819 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.220773 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:44.321589 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.321551 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:44.506239 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.506150 2560 generic.go:356] "Generic (PLEG): container finished" podID="587f3ae3-56fa-4ece-8580-f9787cdc0db4" containerID="8e46c9e3632ec2a158a2d1401a62c8ddf61ced4a769594148f1d10e82c93a452" exitCode=0 Sep 18 17:49:44.506239 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.506198 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" Sep 18 17:49:44.506239 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.506213 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" event={"ID":"587f3ae3-56fa-4ece-8580-f9787cdc0db4","Type":"ContainerDied","Data":"8e46c9e3632ec2a158a2d1401a62c8ddf61ced4a769594148f1d10e82c93a452"} Sep 18 17:49:44.506496 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.506256 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp" event={"ID":"587f3ae3-56fa-4ece-8580-f9787cdc0db4","Type":"ContainerDied","Data":"d37804ae228cc699692238ef10e1dfef4369619058fa5ebadd4b802b9a9b57f6"} Sep 18 17:49:44.506496 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.506278 2560 scope.go:117] "RemoveContainer" containerID="8e46c9e3632ec2a158a2d1401a62c8ddf61ced4a769594148f1d10e82c93a452" Sep 18 17:49:44.516958 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.516939 2560 scope.go:117] "RemoveContainer" containerID="8e46c9e3632ec2a158a2d1401a62c8ddf61ced4a769594148f1d10e82c93a452" Sep 18 17:49:44.517280 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:49:44.517261 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e46c9e3632ec2a158a2d1401a62c8ddf61ced4a769594148f1d10e82c93a452\": container with ID starting with 8e46c9e3632ec2a158a2d1401a62c8ddf61ced4a769594148f1d10e82c93a452 not found: ID does not exist" containerID="8e46c9e3632ec2a158a2d1401a62c8ddf61ced4a769594148f1d10e82c93a452" Sep 18 17:49:44.517375 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.517314 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e46c9e3632ec2a158a2d1401a62c8ddf61ced4a769594148f1d10e82c93a452"} err="failed to get container status \"8e46c9e3632ec2a158a2d1401a62c8ddf61ced4a769594148f1d10e82c93a452\": rpc error: code = NotFound desc = could not find container \"8e46c9e3632ec2a158a2d1401a62c8ddf61ced4a769594148f1d10e82c93a452\": container with ID starting with 8e46c9e3632ec2a158a2d1401a62c8ddf61ced4a769594148f1d10e82c93a452 not found: ID does not exist" Sep 18 17:49:44.529457 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.529422 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp"] Sep 18 17:49:44.533851 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.533817 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-808470-54b675cd48-wxkwp"] Sep 18 17:49:44.864874 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:44.864775 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="587f3ae3-56fa-4ece-8580-f9787cdc0db4" path="/var/lib/kubelet/pods/587f3ae3-56fa-4ece-8580-f9787cdc0db4/volumes" Sep 18 17:49:46.410895 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:46.410857 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:49:49.036680 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:49.036650 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:49:49.042558 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:49.042526 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:49:49.042740 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:49.042723 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:49:49.049306 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:49.049276 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:49:50.662622 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:50.662582 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:50.756249 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:50.756216 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:50.867160 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:50.867134 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66"] Sep 18 17:49:50.867389 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:50.867365 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerName="istio-proxy" containerID="cri-o://deedd1d11afe08fcbe960f6a0708d837d7d9f15c0a0f5a8abba30fdf4321c14e" gracePeriod=30 Sep 18 17:49:50.883206 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:50.883168 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:49:56.131874 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.131810 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:49:56.265813 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.265777 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-envoy\") pod \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " Sep 18 17:49:56.265813 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.265821 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-workload-socket\") pod \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " Sep 18 17:49:56.266098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.265866 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-workload-certs\") pod \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " Sep 18 17:49:56.266098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.265924 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9js9\" (UniqueName: \"kubernetes.io/projected/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-kube-api-access-w9js9\") pod \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " Sep 18 17:49:56.266098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.265981 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-credential-socket\") pod \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " Sep 18 17:49:56.266098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.266023 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-data\") pod \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " Sep 18 17:49:56.266098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.266046 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-token\") pod \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " Sep 18 17:49:56.266098 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.266087 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-podinfo\") pod \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " Sep 18 17:49:56.266445 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.266113 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istiod-ca-cert\") pod \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\" (UID: \"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9\") " Sep 18 17:49:56.266445 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.266136 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" (UID: "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:56.266445 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.266261 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" (UID: "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:56.266445 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.266419 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" (UID: "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:56.266641 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.266505 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-data" (OuterVolumeSpecName: "istio-data") pod "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" (UID: "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:56.266641 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.266606 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-credential-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:56.266641 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.266628 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-data\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:56.266641 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.266642 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-workload-socket\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:56.266809 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.266654 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-workload-certs\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:56.266809 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.266792 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" (UID: "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 18 17:49:56.268572 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.268548 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" (UID: "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:49:56.268663 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.268569 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-token" (OuterVolumeSpecName: "istio-token") pod "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" (UID: "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:49:56.268756 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.268737 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-kube-api-access-w9js9" (OuterVolumeSpecName: "kube-api-access-w9js9") pod "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" (UID: "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9"). InnerVolumeSpecName "kube-api-access-w9js9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:49:56.268807 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.268737 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" (UID: "45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 18 17:49:56.367283 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.367248 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-token\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:56.367283 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.367277 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-podinfo\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:56.367283 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.367285 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istiod-ca-cert\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:56.367543 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.367319 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-istio-envoy\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:56.367543 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.367332 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9js9\" (UniqueName: \"kubernetes.io/projected/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9-kube-api-access-w9js9\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:49:56.573150 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.573112 2560 generic.go:356] "Generic (PLEG): container finished" podID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerID="deedd1d11afe08fcbe960f6a0708d837d7d9f15c0a0f5a8abba30fdf4321c14e" exitCode=0 Sep 18 17:49:56.573345 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.573154 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" event={"ID":"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9","Type":"ContainerDied","Data":"deedd1d11afe08fcbe960f6a0708d837d7d9f15c0a0f5a8abba30fdf4321c14e"} Sep 18 17:49:56.573345 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.573161 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" Sep 18 17:49:56.573345 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.573207 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66" event={"ID":"45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9","Type":"ContainerDied","Data":"330bc4176c2b35fe539464904262fa7b151c3c02fa327f2226dd8c79d0292404"} Sep 18 17:49:56.573345 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.573231 2560 scope.go:117] "RemoveContainer" containerID="deedd1d11afe08fcbe960f6a0708d837d7d9f15c0a0f5a8abba30fdf4321c14e" Sep 18 17:49:56.582808 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.582793 2560 scope.go:117] "RemoveContainer" containerID="2c3dd83cc2c5bbf4b8da151bce6862f4bf5cf8a4b678e3336f695637b917f178" Sep 18 17:49:56.590865 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.590849 2560 scope.go:117] "RemoveContainer" containerID="deedd1d11afe08fcbe960f6a0708d837d7d9f15c0a0f5a8abba30fdf4321c14e" Sep 18 17:49:56.591143 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:49:56.591121 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deedd1d11afe08fcbe960f6a0708d837d7d9f15c0a0f5a8abba30fdf4321c14e\": container with ID starting with deedd1d11afe08fcbe960f6a0708d837d7d9f15c0a0f5a8abba30fdf4321c14e not found: ID does not exist" containerID="deedd1d11afe08fcbe960f6a0708d837d7d9f15c0a0f5a8abba30fdf4321c14e" Sep 18 17:49:56.591181 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.591152 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deedd1d11afe08fcbe960f6a0708d837d7d9f15c0a0f5a8abba30fdf4321c14e"} err="failed to get container status \"deedd1d11afe08fcbe960f6a0708d837d7d9f15c0a0f5a8abba30fdf4321c14e\": rpc error: code = NotFound desc = could not find container \"deedd1d11afe08fcbe960f6a0708d837d7d9f15c0a0f5a8abba30fdf4321c14e\": container with ID starting with deedd1d11afe08fcbe960f6a0708d837d7d9f15c0a0f5a8abba30fdf4321c14e not found: ID does not exist" Sep 18 17:49:56.591181 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.591169 2560 scope.go:117] "RemoveContainer" containerID="2c3dd83cc2c5bbf4b8da151bce6862f4bf5cf8a4b678e3336f695637b917f178" Sep 18 17:49:56.591445 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:49:56.591428 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3dd83cc2c5bbf4b8da151bce6862f4bf5cf8a4b678e3336f695637b917f178\": container with ID starting with 2c3dd83cc2c5bbf4b8da151bce6862f4bf5cf8a4b678e3336f695637b917f178 not found: ID does not exist" containerID="2c3dd83cc2c5bbf4b8da151bce6862f4bf5cf8a4b678e3336f695637b917f178" Sep 18 17:49:56.591492 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.591451 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3dd83cc2c5bbf4b8da151bce6862f4bf5cf8a4b678e3336f695637b917f178"} err="failed to get container status \"2c3dd83cc2c5bbf4b8da151bce6862f4bf5cf8a4b678e3336f695637b917f178\": rpc error: code = NotFound desc = could not find container \"2c3dd83cc2c5bbf4b8da151bce6862f4bf5cf8a4b678e3336f695637b917f178\": container with ID starting with 2c3dd83cc2c5bbf4b8da151bce6862f4bf5cf8a4b678e3336f695637b917f178 not found: ID does not exist" Sep 18 17:49:56.601075 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.601045 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66"] Sep 18 17:49:56.608189 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.608166 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-808470-openshift-default-6bd756bddd-kpm66"] Sep 18 17:49:56.864913 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:56.864834 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" path="/var/lib/kubelet/pods/45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9/volumes" Sep 18 17:49:57.026724 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:49:57.026694 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:50:28.779921 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:50:28.779885 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:50:44.982619 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:50:44.982547 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:50:54.907016 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:50:54.906984 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:51:16.714417 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:51:16.714382 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:51:17.340683 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:51:17.340649 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:51:18.826141 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:51:18.826103 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:51:49.307196 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:51:49.307165 2560 scope.go:117] "RemoveContainer" containerID="72c7e81d89141de727616a17d5c78370fb82d1534fc4e2d8cccf50d76781fa4e" Sep 18 17:51:56.910709 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:51:56.910666 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:52:05.152549 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:05.152461 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:52:13.018916 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:13.018882 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:52:31.320690 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:31.320655 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:52:38.012750 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:38.012712 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:52:41.024445 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:41.024405 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9"] Sep 18 17:52:41.024858 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:41.024677 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" podUID="7e88128b-b901-4a7d-905d-53641faa452b" containerName="manager" containerID="cri-o://517bc0b2db8c3a611d7bb0b5c95279ae1b2831699f9619df2b020096cb1b1df5" gracePeriod=10 Sep 18 17:52:41.888796 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:41.888766 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:52:41.972085 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:41.971997 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e88128b-b901-4a7d-905d-53641faa452b-extensions-socket-volume\") pod \"7e88128b-b901-4a7d-905d-53641faa452b\" (UID: \"7e88128b-b901-4a7d-905d-53641faa452b\") " Sep 18 17:52:41.972274 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:41.972187 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29xzk\" (UniqueName: \"kubernetes.io/projected/7e88128b-b901-4a7d-905d-53641faa452b-kube-api-access-29xzk\") pod \"7e88128b-b901-4a7d-905d-53641faa452b\" (UID: \"7e88128b-b901-4a7d-905d-53641faa452b\") " Sep 18 17:52:41.972419 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:41.972397 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e88128b-b901-4a7d-905d-53641faa452b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7e88128b-b901-4a7d-905d-53641faa452b" (UID: "7e88128b-b901-4a7d-905d-53641faa452b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 18 17:52:41.972502 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:41.972490 2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e88128b-b901-4a7d-905d-53641faa452b-extensions-socket-volume\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:52:41.974266 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:41.974241 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e88128b-b901-4a7d-905d-53641faa452b-kube-api-access-29xzk" (OuterVolumeSpecName: "kube-api-access-29xzk") pod "7e88128b-b901-4a7d-905d-53641faa452b" (UID: "7e88128b-b901-4a7d-905d-53641faa452b"). InnerVolumeSpecName "kube-api-access-29xzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 18 17:52:42.073525 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.073490 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29xzk\" (UniqueName: \"kubernetes.io/projected/7e88128b-b901-4a7d-905d-53641faa452b-kube-api-access-29xzk\") on node \"ip-10-0-128-9.ec2.internal\" DevicePath \"\"" Sep 18 17:52:42.460913 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.460877 2560 generic.go:356] "Generic (PLEG): container finished" podID="7e88128b-b901-4a7d-905d-53641faa452b" containerID="517bc0b2db8c3a611d7bb0b5c95279ae1b2831699f9619df2b020096cb1b1df5" exitCode=0 Sep 18 17:52:42.461104 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.460920 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" Sep 18 17:52:42.461104 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.460945 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" event={"ID":"7e88128b-b901-4a7d-905d-53641faa452b","Type":"ContainerDied","Data":"517bc0b2db8c3a611d7bb0b5c95279ae1b2831699f9619df2b020096cb1b1df5"} Sep 18 17:52:42.461104 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.460992 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9" event={"ID":"7e88128b-b901-4a7d-905d-53641faa452b","Type":"ContainerDied","Data":"1108b7e7549cf015bc49ab69e5596c409afeec200de23aaa050b38c7bcc055b2"} Sep 18 17:52:42.461104 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.461015 2560 scope.go:117] "RemoveContainer" containerID="517bc0b2db8c3a611d7bb0b5c95279ae1b2831699f9619df2b020096cb1b1df5" Sep 18 17:52:42.470673 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.470655 2560 scope.go:117] "RemoveContainer" containerID="347ca721606971b4f16d7fd727f0a748e2f20c9fe0a942ed09a1e8a11920d682" Sep 18 17:52:42.478935 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.478916 2560 scope.go:117] "RemoveContainer" containerID="517bc0b2db8c3a611d7bb0b5c95279ae1b2831699f9619df2b020096cb1b1df5" Sep 18 17:52:42.479248 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:52:42.479226 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517bc0b2db8c3a611d7bb0b5c95279ae1b2831699f9619df2b020096cb1b1df5\": container with ID starting with 517bc0b2db8c3a611d7bb0b5c95279ae1b2831699f9619df2b020096cb1b1df5 not found: ID does not exist" containerID="517bc0b2db8c3a611d7bb0b5c95279ae1b2831699f9619df2b020096cb1b1df5" Sep 18 17:52:42.479333 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.479255 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517bc0b2db8c3a611d7bb0b5c95279ae1b2831699f9619df2b020096cb1b1df5"} err="failed to get container status \"517bc0b2db8c3a611d7bb0b5c95279ae1b2831699f9619df2b020096cb1b1df5\": rpc error: code = NotFound desc = could not find container \"517bc0b2db8c3a611d7bb0b5c95279ae1b2831699f9619df2b020096cb1b1df5\": container with ID starting with 517bc0b2db8c3a611d7bb0b5c95279ae1b2831699f9619df2b020096cb1b1df5 not found: ID does not exist" Sep 18 17:52:42.479333 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.479271 2560 scope.go:117] "RemoveContainer" containerID="347ca721606971b4f16d7fd727f0a748e2f20c9fe0a942ed09a1e8a11920d682" Sep 18 17:52:42.479562 ip-10-0-128-9 kubenswrapper[2560]: E0918 17:52:42.479543 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347ca721606971b4f16d7fd727f0a748e2f20c9fe0a942ed09a1e8a11920d682\": container with ID starting with 347ca721606971b4f16d7fd727f0a748e2f20c9fe0a942ed09a1e8a11920d682 not found: ID does not exist" containerID="347ca721606971b4f16d7fd727f0a748e2f20c9fe0a942ed09a1e8a11920d682" Sep 18 17:52:42.479600 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.479566 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347ca721606971b4f16d7fd727f0a748e2f20c9fe0a942ed09a1e8a11920d682"} err="failed to get container status \"347ca721606971b4f16d7fd727f0a748e2f20c9fe0a942ed09a1e8a11920d682\": rpc error: code = NotFound desc = could not find container \"347ca721606971b4f16d7fd727f0a748e2f20c9fe0a942ed09a1e8a11920d682\": container with ID starting with 347ca721606971b4f16d7fd727f0a748e2f20c9fe0a942ed09a1e8a11920d682 not found: ID does not exist" Sep 18 17:52:42.485732 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.485705 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9"] Sep 18 17:52:42.489498 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.489472 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-md8r9"] Sep 18 17:52:42.865135 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:52:42.865040 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e88128b-b901-4a7d-905d-53641faa452b" path="/var/lib/kubelet/pods/7e88128b-b901-4a7d-905d-53641faa452b/volumes" Sep 18 17:53:24.419008 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:24.418968 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:53:24.508001 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:24.507957 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:53:41.763700 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.763622 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn"] Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764035 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerName="istio-proxy" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764047 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerName="istio-proxy" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764061 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerName="istio-proxy" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764067 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerName="istio-proxy" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764074 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerName="istio-proxy" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764080 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerName="istio-proxy" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764090 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="587f3ae3-56fa-4ece-8580-f9787cdc0db4" containerName="maas-api" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764095 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="587f3ae3-56fa-4ece-8580-f9787cdc0db4" containerName="maas-api" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764104 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerName="istio-proxy" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764109 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerName="istio-proxy" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764119 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e88128b-b901-4a7d-905d-53641faa452b" containerName="manager" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764124 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e88128b-b901-4a7d-905d-53641faa452b" containerName="manager" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764131 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerName="istio-proxy" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764136 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerName="istio-proxy" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764149 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42493925-5c68-4aad-916d-47abab32b187" containerName="istio-proxy" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764154 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="42493925-5c68-4aad-916d-47abab32b187" containerName="istio-proxy" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764165 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972da268-f047-46c4-a566-ad813b1a0687" containerName="maas-api" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764171 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="972da268-f047-46c4-a566-ad813b1a0687" containerName="maas-api" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764177 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42493925-5c68-4aad-916d-47abab32b187" containerName="istio-proxy" Sep 18 17:53:41.764176 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764182 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="42493925-5c68-4aad-916d-47abab32b187" containerName="istio-proxy" Sep 18 17:53:41.764810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764241 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="972da268-f047-46c4-a566-ad813b1a0687" containerName="maas-api" Sep 18 17:53:41.764810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764249 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerName="istio-proxy" Sep 18 17:53:41.764810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764255 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerName="istio-proxy" Sep 18 17:53:41.764810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764262 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e88128b-b901-4a7d-905d-53641faa452b" containerName="manager" Sep 18 17:53:41.764810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764268 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="42493925-5c68-4aad-916d-47abab32b187" containerName="istio-proxy" Sep 18 17:53:41.764810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764272 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerName="istio-proxy" Sep 18 17:53:41.764810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764278 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e88128b-b901-4a7d-905d-53641faa452b" containerName="manager" Sep 18 17:53:41.764810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764283 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="42493925-5c68-4aad-916d-47abab32b187" containerName="istio-proxy" Sep 18 17:53:41.764810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764288 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="587f3ae3-56fa-4ece-8580-f9787cdc0db4" containerName="maas-api" Sep 18 17:53:41.764810 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.764308 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerName="istio-proxy" Sep 18 17:53:41.767161 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.767138 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn" Sep 18 17:53:41.771340 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.771319 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v5c59\"" Sep 18 17:53:41.790087 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.790058 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn"] Sep 18 17:53:41.856393 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.856356 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w7c7\" (UniqueName: \"kubernetes.io/projected/52bba90d-a5df-41ce-bd59-c02e7edaac48-kube-api-access-8w7c7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ndxqn\" (UID: \"52bba90d-a5df-41ce-bd59-c02e7edaac48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn" Sep 18 17:53:41.856552 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.856423 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52bba90d-a5df-41ce-bd59-c02e7edaac48-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ndxqn\" (UID: \"52bba90d-a5df-41ce-bd59-c02e7edaac48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn" Sep 18 17:53:41.957537 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.957498 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w7c7\" (UniqueName: \"kubernetes.io/projected/52bba90d-a5df-41ce-bd59-c02e7edaac48-kube-api-access-8w7c7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ndxqn\" (UID: \"52bba90d-a5df-41ce-bd59-c02e7edaac48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn" Sep 18 17:53:41.957724 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.957547 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52bba90d-a5df-41ce-bd59-c02e7edaac48-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ndxqn\" (UID: \"52bba90d-a5df-41ce-bd59-c02e7edaac48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn" Sep 18 17:53:41.957974 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.957954 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52bba90d-a5df-41ce-bd59-c02e7edaac48-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ndxqn\" (UID: \"52bba90d-a5df-41ce-bd59-c02e7edaac48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn" Sep 18 17:53:41.967249 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:41.967225 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w7c7\" (UniqueName: \"kubernetes.io/projected/52bba90d-a5df-41ce-bd59-c02e7edaac48-kube-api-access-8w7c7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ndxqn\" (UID: \"52bba90d-a5df-41ce-bd59-c02e7edaac48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn" Sep 18 17:53:42.077379 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:42.077273 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn" Sep 18 17:53:42.367121 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:42.367092 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn"] Sep 18 17:53:42.369952 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:53:42.369919 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bba90d_a5df_41ce_bd59_c02e7edaac48.slice/crio-f95a2e6614a7c4beb77794f5bf2f7c8d6d838d02438863acb72f17378d54e6ab WatchSource:0}: Error finding container f95a2e6614a7c4beb77794f5bf2f7c8d6d838d02438863acb72f17378d54e6ab: Status 404 returned error can't find the container with id f95a2e6614a7c4beb77794f5bf2f7c8d6d838d02438863acb72f17378d54e6ab Sep 18 17:53:42.791358 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:42.791321 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn" event={"ID":"52bba90d-a5df-41ce-bd59-c02e7edaac48","Type":"ContainerStarted","Data":"a4f7a670022e45c408b0d9b972a4fdcf23190501952baaba7c065bea1163933d"} Sep 18 17:53:42.791765 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:42.791381 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn" event={"ID":"52bba90d-a5df-41ce-bd59-c02e7edaac48","Type":"ContainerStarted","Data":"f95a2e6614a7c4beb77794f5bf2f7c8d6d838d02438863acb72f17378d54e6ab"} Sep 18 17:53:42.791765 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:42.791428 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn" Sep 18 17:53:42.817627 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:42.817576 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn" podStartSLOduration=1.8175583309999999 podStartE2EDuration="1.817558331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:53:42.815907948 +0000 UTC m=+2336.069068793" watchObservedRunningTime="2026-09-18 17:53:42.817558331 +0000 UTC m=+2336.070719147" Sep 18 17:53:53.799131 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:53:53.799100 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ndxqn" Sep 18 17:54:10.112429 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:54:10.112391 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:54:18.644109 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:54:18.644065 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:54:39.611708 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:54:39.611669 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:54:40.161658 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:54:40.161620 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:54:42.753450 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:54:42.753380 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:54:49.080221 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:54:49.080194 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:54:49.084501 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:54:49.084479 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:54:49.086548 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:54:49.086523 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:54:49.090546 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:54:49.090528 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qf4rc_17ee44c6-7ddd-473c-8b5b-c2e908141e09/ovn-acl-logging/0.log" Sep 18 17:54:51.478817 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:54:51.478780 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:54:54.244289 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:54:54.244253 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:55:09.084875 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:09.084787 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:55:11.417707 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:11.417665 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:55:11.656076 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:11.656032 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:55:19.635271 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:19.635231 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:55:27.513604 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.513563 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:55:27.570314 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.564009 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj"] Sep 18 17:55:27.571563 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.571510 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerName="istio-proxy" Sep 18 17:55:27.571563 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.571544 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="45f3fb3b-7f0e-4bc5-a5ce-c88804fbdad9" containerName="istio-proxy" Sep 18 17:55:27.571753 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.571594 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e88128b-b901-4a7d-905d-53641faa452b" containerName="manager" Sep 18 17:55:27.571753 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.571604 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e88128b-b901-4a7d-905d-53641faa452b" containerName="manager" Sep 18 17:55:27.571952 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.571926 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94d9e32-57db-4e74-a2bd-99ebb26d9bfd" containerName="istio-proxy" Sep 18 17:55:27.572034 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.571957 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a33da0e8-b61f-4bca-aa2a-afc1cdb5628c" containerName="istio-proxy" Sep 18 17:55:27.576074 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.576050 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:27.580043 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.580005 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj"] Sep 18 17:55:27.582384 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.582360 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 18 17:55:27.582384 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.582375 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 18 17:55:27.583539 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.583511 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 18 17:55:27.583655 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.583615 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 18 17:55:27.583718 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.583662 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sjtq4\"" Sep 18 17:55:27.710833 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.710791 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d952dcb4-88ed-47a5-b4ec-b91ae90a9b65-maas-api-tls\") pod \"maas-api-78bfb4b689-njnnj\" (UID: \"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65\") " pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:27.711032 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.710879 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d952dcb4-88ed-47a5-b4ec-b91ae90a9b65-metrics-certs\") pod \"maas-api-78bfb4b689-njnnj\" (UID: \"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65\") " pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:27.711032 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.710930 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj5dw\" (UniqueName: \"kubernetes.io/projected/d952dcb4-88ed-47a5-b4ec-b91ae90a9b65-kube-api-access-tj5dw\") pod \"maas-api-78bfb4b689-njnnj\" (UID: \"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65\") " pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:27.711032 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.711011 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d952dcb4-88ed-47a5-b4ec-b91ae90a9b65-openshift-service-ca\") pod \"maas-api-78bfb4b689-njnnj\" (UID: \"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65\") " pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:27.812039 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.811929 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d952dcb4-88ed-47a5-b4ec-b91ae90a9b65-openshift-service-ca\") pod \"maas-api-78bfb4b689-njnnj\" (UID: \"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65\") " pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:27.812039 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.811995 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d952dcb4-88ed-47a5-b4ec-b91ae90a9b65-maas-api-tls\") pod \"maas-api-78bfb4b689-njnnj\" (UID: \"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65\") " pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:27.812039 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.812038 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d952dcb4-88ed-47a5-b4ec-b91ae90a9b65-metrics-certs\") pod \"maas-api-78bfb4b689-njnnj\" (UID: \"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65\") " pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:27.812442 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.812070 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj5dw\" (UniqueName: \"kubernetes.io/projected/d952dcb4-88ed-47a5-b4ec-b91ae90a9b65-kube-api-access-tj5dw\") pod \"maas-api-78bfb4b689-njnnj\" (UID: \"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65\") " pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:27.812726 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.812702 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d952dcb4-88ed-47a5-b4ec-b91ae90a9b65-openshift-service-ca\") pod \"maas-api-78bfb4b689-njnnj\" (UID: \"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65\") " pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:27.814614 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.814583 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d952dcb4-88ed-47a5-b4ec-b91ae90a9b65-maas-api-tls\") pod \"maas-api-78bfb4b689-njnnj\" (UID: \"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65\") " pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:27.814708 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.814641 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d952dcb4-88ed-47a5-b4ec-b91ae90a9b65-metrics-certs\") pod \"maas-api-78bfb4b689-njnnj\" (UID: \"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65\") " pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:27.821094 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.821070 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj5dw\" (UniqueName: \"kubernetes.io/projected/d952dcb4-88ed-47a5-b4ec-b91ae90a9b65-kube-api-access-tj5dw\") pod \"maas-api-78bfb4b689-njnnj\" (UID: \"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65\") " pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:27.889038 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:27.888995 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:28.169449 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:28.169419 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj"] Sep 18 17:55:28.171475 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:55:28.171443 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd952dcb4_88ed_47a5_b4ec_b91ae90a9b65.slice/crio-046515d3bce2b40468f2b5f86c8f3cee7bd18bd6145776b6bc082824e52a7781 WatchSource:0}: Error finding container 046515d3bce2b40468f2b5f86c8f3cee7bd18bd6145776b6bc082824e52a7781: Status 404 returned error can't find the container with id 046515d3bce2b40468f2b5f86c8f3cee7bd18bd6145776b6bc082824e52a7781 Sep 18 17:55:28.172954 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:28.172932 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 18 17:55:28.345961 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:28.345917 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" event={"ID":"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65","Type":"ContainerStarted","Data":"046515d3bce2b40468f2b5f86c8f3cee7bd18bd6145776b6bc082824e52a7781"} Sep 18 17:55:29.352655 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:29.352613 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" event={"ID":"d952dcb4-88ed-47a5-b4ec-b91ae90a9b65","Type":"ContainerStarted","Data":"470d2190f7eba132663be4554f00c24363fef4744b823591c0cbaeb5d5da88f3"} Sep 18 17:55:29.353059 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:29.352683 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:29.374325 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:29.374262 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" podStartSLOduration=1.953794163 podStartE2EDuration="2.374243994s" totalImagesPullingTime="420.449831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:55:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:55:29.37202208 +0000 UTC m=+2442.625182898" watchObservedRunningTime="2026-09-18 17:55:29.374243994 +0000 UTC m=+2442.627404811" Sep 18 17:55:35.277279 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:35.277244 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:55:35.363199 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:35.363170 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78bfb4b689-njnnj" Sep 18 17:55:42.519347 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:42.519308 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:55:50.947476 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:50.947435 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:55:57.485110 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:57.485067 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:55:58.215721 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:55:58.215684 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-rvxk8"] Sep 18 17:56:06.976503 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:06.976471 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zzn5f_281a8253-e792-4359-8464-823a81f61dc0/manager/0.log" Sep 18 17:56:07.095240 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:07.095207 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jhblt_c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d/manager/0.log" Sep 18 17:56:07.810133 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:07.810103 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9496889b9-g6qzj_9ecd4364-2a82-4806-8853-a6d9240e06c6/copy-manifests/0.log" Sep 18 17:56:07.828999 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:07.828965 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9496889b9-g6qzj_9ecd4364-2a82-4806-8853-a6d9240e06c6/manager/0.log" Sep 18 17:56:08.251659 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:08.251628 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78bfb4b689-njnnj_d952dcb4-88ed-47a5-b4ec-b91ae90a9b65/maas-api/0.log" Sep 18 17:56:09.587156 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:09.587122 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-lkjhw_7d3e0259-5dea-4114-a8ab-f4206d0d260c/manager/0.log" Sep 18 17:56:09.813264 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:09.813230 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-ndxqn_52bba90d-a5df-41ce-bd59-c02e7edaac48/manager/0.log" Sep 18 17:56:09.933434 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:09.933404 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-rvxk8_d49821d9-884c-4e55-bea6-ca95b943e2e5/limitador/0.log" Sep 18 17:56:10.057583 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:10.057552 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-6blrb_9ad315f6-9ff4-4a1a-a73d-32e66a02dd82/manager/0.log" Sep 18 17:56:10.396356 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:10.396256 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj_96c39c3a-28d7-483f-9e54-6ba0eec7b333/istio-proxy/0.log" Sep 18 17:56:11.794586 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:11.794553 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-mgmv7_2126dbfa-e868-42d2-8a36-0095b5de94fc/main/0.log" Sep 18 17:56:18.134895 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:18.134855 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:56:18.298485 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:18.298457 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" limit=1024 Sep 18 17:56:21.630575 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:21.630543 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qrbrr_af831002-a9cc-4239-9821-98618e18e764/global-pull-secret-syncer/0.log" Sep 18 17:56:21.771339 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:21.771286 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-78r8w_156fc6c4-1430-428e-aa27-fb144f9ab893/konnectivity-agent/0.log" Sep 18 17:56:21.957018 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:21.956984 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-9.ec2.internal_e55b801a498c78c600fd5d152605bab9/haproxy/0.log" Sep 18 17:56:26.295459 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:26.295426 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-lkjhw_7d3e0259-5dea-4114-a8ab-f4206d0d260c/manager/0.log" Sep 18 17:56:26.379350 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:26.379314 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-ndxqn_52bba90d-a5df-41ce-bd59-c02e7edaac48/manager/0.log" Sep 18 17:56:26.404879 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:26.404848 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-rvxk8_d49821d9-884c-4e55-bea6-ca95b943e2e5/limitador/0.log" Sep 18 17:56:26.498727 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:26.498643 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-6blrb_9ad315f6-9ff4-4a1a-a73d-32e66a02dd82/manager/0.log" Sep 18 17:56:28.021540 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.021512 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1/alertmanager/0.log" Sep 18 17:56:28.045930 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.045905 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1/config-reloader/0.log" Sep 18 17:56:28.070308 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.070253 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1/kube-rbac-proxy-web/0.log" Sep 18 17:56:28.092309 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.092280 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1/kube-rbac-proxy/0.log" Sep 18 17:56:28.124567 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.124536 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1/kube-rbac-proxy-metric/0.log" Sep 18 17:56:28.146353 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.146326 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1/prom-label-proxy/0.log" Sep 18 17:56:28.171101 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.171055 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c73bc2c-eeaa-4aa5-9473-0cadc8a7c6a1/init-config-reloader/0.log" Sep 18 17:56:28.223112 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.223079 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-4xbwf_a4216580-a12b-49e1-b2e1-607171a27a96/cluster-monitoring-operator/0.log" Sep 18 17:56:28.274570 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.274501 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-d2mw7_09d4fbe4-81ee-480e-a266-c1f65e06b8ff/kube-state-metrics/0.log" Sep 18 17:56:28.295239 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.295211 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-d2mw7_09d4fbe4-81ee-480e-a266-c1f65e06b8ff/kube-rbac-proxy-main/0.log" Sep 18 17:56:28.319357 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.319328 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-d2mw7_09d4fbe4-81ee-480e-a266-c1f65e06b8ff/kube-rbac-proxy-self/0.log" Sep 18 17:56:28.345553 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.345530 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5d4c6d666-2lpbp_a42baf8a-7f22-47bd-9d1f-a658bc38135d/metrics-server/0.log" Sep 18 17:56:28.372198 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.372164 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-w77st_6e7287c8-edd4-45d6-a36f-1b53d0f2172e/monitoring-plugin/0.log" Sep 18 17:56:28.569626 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.569597 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gdwtp_183d465b-7662-46b0-b2d7-9c5a6fc9f82a/node-exporter/0.log" Sep 18 17:56:28.595513 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.595486 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gdwtp_183d465b-7662-46b0-b2d7-9c5a6fc9f82a/kube-rbac-proxy/0.log" Sep 18 17:56:28.619013 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.618990 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gdwtp_183d465b-7662-46b0-b2d7-9c5a6fc9f82a/init-textfile/0.log" Sep 18 17:56:28.649568 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.649534 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-qmv9c_2df05aae-dbe0-4100-a2ba-da02010f7dd9/kube-rbac-proxy-main/0.log" Sep 18 17:56:28.675750 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.675722 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-qmv9c_2df05aae-dbe0-4100-a2ba-da02010f7dd9/kube-rbac-proxy-self/0.log" Sep 18 17:56:28.710453 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.710419 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-qmv9c_2df05aae-dbe0-4100-a2ba-da02010f7dd9/openshift-state-metrics/0.log" Sep 18 17:56:28.762580 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.762556 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd/prometheus/0.log" Sep 18 17:56:28.817095 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.817072 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd/config-reloader/0.log" Sep 18 17:56:28.875085 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:28.875059 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd/thanos-sidecar/0.log" Sep 18 17:56:29.028514 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:29.028480 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd/kube-rbac-proxy-web/0.log" Sep 18 17:56:29.248676 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:29.248647 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd/kube-rbac-proxy/0.log" Sep 18 17:56:29.313163 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:29.313139 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd/kube-rbac-proxy-thanos/0.log" Sep 18 17:56:29.366282 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:29.366256 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fe2fdaa-2ff6-40e2-ba63-0e383bb921cd/init-config-reloader/0.log" Sep 18 17:56:29.500984 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:29.500953 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-nt5qt_8692e40d-87b0-48dc-ae8d-0f9dc1592331/prometheus-operator/0.log" Sep 18 17:56:29.544735 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:29.544701 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-nt5qt_8692e40d-87b0-48dc-ae8d-0f9dc1592331/kube-rbac-proxy/0.log" Sep 18 17:56:29.579076 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:29.579045 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-8stwn_29a6d4d4-6c68-4fde-b929-377ff159b020/prometheus-operator-admission-webhook/0.log" Sep 18 17:56:29.724511 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:29.724487 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f756495df-fzz5b_790f80ee-9474-4c64-842e-7a0b7cc741dc/thanos-query/0.log" Sep 18 17:56:29.750070 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:29.750044 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f756495df-fzz5b_790f80ee-9474-4c64-842e-7a0b7cc741dc/kube-rbac-proxy-web/0.log" Sep 18 17:56:29.778786 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:29.778679 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f756495df-fzz5b_790f80ee-9474-4c64-842e-7a0b7cc741dc/kube-rbac-proxy/0.log" Sep 18 17:56:29.812003 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:29.811974 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f756495df-fzz5b_790f80ee-9474-4c64-842e-7a0b7cc741dc/prom-label-proxy/0.log" Sep 18 17:56:29.844458 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:29.844382 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f756495df-fzz5b_790f80ee-9474-4c64-842e-7a0b7cc741dc/kube-rbac-proxy-rules/0.log" Sep 18 17:56:29.881585 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:29.881560 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f756495df-fzz5b_790f80ee-9474-4c64-842e-7a0b7cc741dc/kube-rbac-proxy-metrics/0.log" Sep 18 17:56:31.501282 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.501247 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262"] Sep 18 17:56:31.505163 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.505144 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.509195 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.509165 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xbl54\"/\"default-dockercfg-222tl\"" Sep 18 17:56:31.509955 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.509935 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xbl54\"/\"openshift-service-ca.crt\"" Sep 18 17:56:31.510227 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.510212 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xbl54\"/\"kube-root-ca.crt\"" Sep 18 17:56:31.517007 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.516982 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262"] Sep 18 17:56:31.624980 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.624945 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d3a2635-8504-4190-8850-306987f07d3b-proc\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.625170 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.625039 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpqrs\" (UniqueName: \"kubernetes.io/projected/0d3a2635-8504-4190-8850-306987f07d3b-kube-api-access-qpqrs\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.625170 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.625069 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d3a2635-8504-4190-8850-306987f07d3b-podres\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.625170 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.625096 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d3a2635-8504-4190-8850-306987f07d3b-lib-modules\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.625283 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.625211 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d3a2635-8504-4190-8850-306987f07d3b-sys\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.726401 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.726356 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d3a2635-8504-4190-8850-306987f07d3b-sys\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.726587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.726423 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d3a2635-8504-4190-8850-306987f07d3b-proc\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.726587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.726494 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d3a2635-8504-4190-8850-306987f07d3b-sys\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.726587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.726517 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpqrs\" (UniqueName: \"kubernetes.io/projected/0d3a2635-8504-4190-8850-306987f07d3b-kube-api-access-qpqrs\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.726587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.726499 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d3a2635-8504-4190-8850-306987f07d3b-proc\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.726587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.726551 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d3a2635-8504-4190-8850-306987f07d3b-podres\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.726587 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.726576 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d3a2635-8504-4190-8850-306987f07d3b-lib-modules\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.726806 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.726675 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d3a2635-8504-4190-8850-306987f07d3b-podres\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.726806 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.726722 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d3a2635-8504-4190-8850-306987f07d3b-lib-modules\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.736601 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.736566 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpqrs\" (UniqueName: \"kubernetes.io/projected/0d3a2635-8504-4190-8850-306987f07d3b-kube-api-access-qpqrs\") pod \"perf-node-gather-daemonset-b5262\" (UID: \"0d3a2635-8504-4190-8850-306987f07d3b\") " pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.815497 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.815404 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:31.938509 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.938480 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/1.log" Sep 18 17:56:31.946268 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:31.946237 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-dp86w_5aa5aa57-bbfa-461a-b12a-e32ad883c338/console-operator/2.log" Sep 18 17:56:32.109050 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:32.108966 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262"] Sep 18 17:56:32.111657 ip-10-0-128-9 kubenswrapper[2560]: W0918 17:56:32.111625 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0d3a2635_8504_4190_8850_306987f07d3b.slice/crio-ce406092d43859ba6375a0cfc2aa3b85b94b343cc7c78ad0eda21456b67fc2f7 WatchSource:0}: Error finding container ce406092d43859ba6375a0cfc2aa3b85b94b343cc7c78ad0eda21456b67fc2f7: Status 404 returned error can't find the container with id ce406092d43859ba6375a0cfc2aa3b85b94b343cc7c78ad0eda21456b67fc2f7 Sep 18 17:56:32.662127 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:32.662098 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65c9bfcd84-wv27g_4f1d5fe1-b9d0-4a77-8ec1-63e5061b2dcd/console/0.log" Sep 18 17:56:32.707556 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:32.707523 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" event={"ID":"0d3a2635-8504-4190-8850-306987f07d3b","Type":"ContainerStarted","Data":"e3053479010efba5d8a16ef4b455ed1ec9df9d641ecd894816a436b706a6e997"} Sep 18 17:56:32.707749 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:32.707577 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" event={"ID":"0d3a2635-8504-4190-8850-306987f07d3b","Type":"ContainerStarted","Data":"ce406092d43859ba6375a0cfc2aa3b85b94b343cc7c78ad0eda21456b67fc2f7"} Sep 18 17:56:32.707749 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:32.707605 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:32.724779 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:32.724744 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-d2c2h_693f2b8e-0aad-43cd-8e2c-d5f9cd57b949/download-server/0.log" Sep 18 17:56:32.730963 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:32.730917 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" podStartSLOduration=1.7309025390000001 podStartE2EDuration="1.730902539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:56:32.73083135 +0000 UTC m=+2505.983992186" watchObservedRunningTime="2026-09-18 17:56:32.730902539 +0000 UTC m=+2505.984063356" Sep 18 17:56:34.603212 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:34.603183 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qqglr_ddf535f2-88b7-40b0-8d49-c9bfe97b3679/dns/0.log" Sep 18 17:56:34.629307 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:34.629270 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qqglr_ddf535f2-88b7-40b0-8d49-c9bfe97b3679/kube-rbac-proxy/0.log" Sep 18 17:56:34.786802 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:34.786774 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4vxw9_4a44888c-a9bf-4378-8785-26bc77ef43bb/dns-node-resolver/0.log" Sep 18 17:56:35.768147 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:35.768117 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-44qw2_b51a62b4-6c7c-4e29-ae8c-4f22d4d66f73/node-ca/0.log" Sep 18 17:56:37.064090 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:37.064059 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj_96c39c3a-28d7-483f-9e54-6ba0eec7b333/istio-proxy/0.log" Sep 18 17:56:38.723256 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:38.723225 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xbl54/perf-node-gather-daemonset-b5262" Sep 18 17:56:38.891863 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:38.891829 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k8tvl_e62a1939-824b-49c5-857a-c0c41fb3c31a/serve-healthcheck-canary/0.log" Sep 18 17:56:39.912368 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:39.912342 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s9kbh_6c6aba87-34e8-41a8-9157-96b93bd9dbda/kube-rbac-proxy/0.log" Sep 18 17:56:39.941578 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:39.941547 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s9kbh_6c6aba87-34e8-41a8-9157-96b93bd9dbda/exporter/0.log" Sep 18 17:56:39.980716 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:39.980686 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s9kbh_6c6aba87-34e8-41a8-9157-96b93bd9dbda/extractor/0.log" Sep 18 17:56:41.426976 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:41.426946 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj_96c39c3a-28d7-483f-9e54-6ba0eec7b333/istio-proxy/0.log" Sep 18 17:56:42.641442 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:42.641407 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zzn5f_281a8253-e792-4359-8464-823a81f61dc0/manager/0.log" Sep 18 17:56:43.078568 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:43.078537 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jhblt_c69e0dab-9ad4-4d1c-9fdf-4c7fc0e28d5d/manager/0.log" Sep 18 17:56:43.489420 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:43.489390 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-p8clj_96c39c3a-28d7-483f-9e54-6ba0eec7b333/istio-proxy/0.log" Sep 18 17:56:43.819011 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:43.818927 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9496889b9-g6qzj_9ecd4364-2a82-4806-8853-a6d9240e06c6/manager/0.log" Sep 18 17:56:43.904391 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:43.904365 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9496889b9-g6qzj_9ecd4364-2a82-4806-8853-a6d9240e06c6/copy-manifests/0.log" Sep 18 17:56:45.274159 ip-10-0-128-9 kubenswrapper[2560]: I0918 17:56:45.274132 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b67bbc446-dmb2k_04323ef3-f6a8-4414-af55-9ddc73a25ab4/manager/0.log"